Hidden Signals Challenge

Department of Homeland Security (DHS)

Summary

The Hidden Signals Challenge was a $300,000 prize competition conducted by the DHS Science & Technology Directorate (S&T) in collaboration with the Office of Health Affairs National Biosurveillance Integration Center. DHS S&T asked innovators from a wide variety of fields—from data science and civic tech to epidemiology—to develop concepts for novel uses of existing data that will identify signals and achieve timelier alerts for biothreats in our cities and communities.

Biothreats occur when harmful pathogens are naturally or deliberately released, posing a risk to national security and public health. Some pathogens can be transmitted from person to person through inhalation or ingestion and from exposure to powders, liquids, or aerosols. Infections caused by biothreats can lead to respiratory distress, gastrointestinal issues, and animal and human deaths. It's often difficult to immediately identify biothreats and contain their spread. This makes early warnings critical.

If and when a biothreat incident occurs, every minute counts. Local and national officials must work together to assess the level of risk, develop an action plan, and intervene. They use a variety of systems and tools that mostly rely on health data to detect signals. There is now an unprecedented opportunity due to the increasing proliferation of new technologies and data sources, including the wealth of open data generated by progressively "smarter" cities and trends observed through aggregation of individual sharing. By harnessing these new streams of information, we may be able to identify and resolve emerging problems with greater speed and confidence.

As part of the Biodetection and Biosurveillance Systems Programs, DHS S&T identified that current detection and surveillance methods for biological threats—both intentional and unintentional releases of biological agents and infectious diseases—are not timely enough to enable early warning and intervention. Extensive literature reviews and interviews with subject matter experts defined numerous problems with existing biosurveillance systems and efforts including:

Barriers to access of relevant situational and health data (e.g., electronic health records)
Lack of confidence in data sources
Uncertainty about existing, evolving, and emerging biological threats
Absence of infrastructure, technologies, policies, and knowledge needed to effectively collect and derive insights from data

DHS intends for this work to be the first step in the design of a local and/or national-level system that could enable city-level operators to make critical and proactive decisions based on the most relevant and actionable insights. To form the basis for a proof of concept, the challenge focused on large metropolitan areas such as New York, Los Angeles, Washington D.C., Chicago, Boston, and Atlanta.

The challenge launched Oct. 17, 2017. DHS S&T used an interagency agreement with NASA's Center of Excellence for Collaborative Innovation to contract with a third-party prize administrator, Luminary Labs. A website was established that provided access to additional information, newsletters, blogs, and other information about the challenge.

Results

The challenge received 37 submissions from entities, teams, and individuals from academia; artificial intelligence / machine learning and information technology fields; city and federal government; health technology and research fields; and research think tanks.

Stage 1 finalists:

Readiness Acceleration & Innovation Network (RAIN), Tacoma, Wash.: Commuter Pattern Analysis for Early Biothreat Detection, a system that cross-references de-identified traffic information with existing municipal health data and internet keyword searches. The tool will be developed to recognize commuter absenteeism to flag a possible disease outbreak.
Vituity, Emeryville, Calif.: Monitoring emergency department wait times to detect emergent influenza pandemics, a model that alerts authorities of spikes in emergency room wait times that can be attributed to emergent flu pandemics. The solution sources real-time data from a network of 142 hospitals in 19 states and is updated hourly, allowing agencies to quickly intervene.
William Pilkington & team, Cabarrus County, N.C.: One Health Alert System, a symptoms database that analyzes the Daily Disease Report's top ten symptoms as seen by 43 health care providers in North Carolina. The model flags disease outbreak using textual predictive analytics and accounts for seasonal rates of change.
Computational Epidemiology Lab at Boston Children's Hospital, Boston, Mass.: Pandemic Pulse, a tool that integrates six data streams to detect biothreat signals. First, it alerts agencies using Twitter, Google Search, transportation, news, and HealthMap data of an anomaly in the data stream, then it tracks potential biothreats using live transportation data on Flu Near You.
Daniel B. Neill and Mallory Nobles, Pittsburgh, Pa.: Pre-syndromic Surveillance, a machine learning system that overlays real-time emergency room chief complaint data with social media and news data using the semantic scan, a novel approach to text analysis. The model detects emerging clusters of rare disease cases that do not correspond to known syndrome types.

Winners:

Grand Prize: The Computational Epidemiology Lab at Boston Children's Hospital received the grand prize of $150,000 for their proposed solution Pandemic Pulse. This system provides a dashboard that integrates Twitter and Google Search data with infectious disease monitoring tools, Flu Near You and HealthMap, to detect biothreat signals. The tool uses a tiered evaluation method to filter data based on pathogen category, information source, and transmission mode.
Runner-up: The runner-up solution, Pre-syndromic Surveillance, submitted by Daniel B. Neill and Mallory Nobles of Pittsburgh received $50,000. This system integrates emergency department chief complaints with data from health clinics and social media to discover outbreaks that do not correspond with known illnesses. The team is piloting a working prototype with New York City's Department of Health and Mental Hygiene and other city agencies.

Areas of Excellence

Area of Excellence #1: "Design the Challenge Structure"

The Challenge was designed with two stages, the first to attract novel concepts and the second to develop the top concepts into detailed system designs. Following the open submissions period, judges selected five finalists based on the criteria, and each finalist was awarded $20,000 in cash prizes.

In Stage 2, finalists from Stage 1 further developed their concepts with guidance from expert mentors as they competed for an additional $200,000 in cash prizes. At the end of Stage 2, finalists submitted detailed system designs, which described how concepts from Stage 1 could be implemented.

For Stage 2, the challenge included Virtual Accelerator modules and optional field exercises that encouraged finalists to ground their systems in reality through user research, persona development, and iterative storyboarding. The Virtual Accelerator featured a module that focused entirely on user research, with an expert panel, reading list and hands-on exercises to discuss with mentors.

Many teams cited the Virtual Accelerator as a critical benefit of the learning experience. In fact, the grand prize winning team said: "The information [the Virtual Accelerator mentors provided] on potential end-user workflows and experiences was incredibly informative. We feel these mentor discussions helped to mature our initial design, with better insights into the variability of the needs of end-users."

Overall, the challenge structure accelerated solution development and impact. The sprint development approach proved effective, even in the complex space of biodefense. With eight weeks for concept development and eight weeks for system design development, the challenge proved that the sprint development approach can produce impact (and in many cases, working prototypes) at an accelerated rate.

The submission requirements and time constraints pushed teams to identify and focus on the most mission-critical elements, leading to a balanced approach for near-term and long-term goals.

"Overall I was impressed by how streamlined the challenge process was and impressed by how much effort and thought the submitters put into this without any money upfront," said one DHS judge.

Area of Excellence #2: "Define Evaluation and Judging Process"

For each stage, the judging process relied on carefully chosen criteria that were weighted equally to emphasize the importance of a balanced approach.

Review panelists and judges were selected to represent a range of expertise relevant to the challenge goals and objectives. Ultimately, a panel of seven judges with expertise in bioinformatics, biological defense, epidemiology, and emergency management helped select the grand prize winner and runner-up.

When evaluating Stage 1 entries, review panelists and judges assigned each submission one to five points in each of the six criteria categories (for a total of up to 30 points):

Originality. Presents a novel approach to the problem, and offers creative solutions and unique hypotheses.
Impact. Has the potential to significantly advance current city-level practices and resources for identifying biothreat signals, and simultaneously complements existing resources.
Feasibility. Demonstrates significant potential to rapidly detect patterns with a high degree of confidence, ideally within a day of exposure and no longer than ten days from exposure, and uses technically sound methods that are backed by credible supporting evidence.
Sustainability. Makes use of freely available and/or low-cost data sources that are readily accessible to city-level operators and DHS on a consistent and long-term basis.
Scalability. Has the ability or potential to expand to other geographic areas, or signals indicative of biothreat incidents or other scenarios of concern for homeland security.
Team. Demonstrates an appropriate level of experience, commitment, and ability to move from concept to system design within the timeline of the Challenge.

When evaluating Stage 2 entries, judges again assigned each submission one to five points in six criteria categories (for a total of up to 30 points):

Empathy. Extent to which the system demonstrates an understanding of and is clearly designed to support a city-level and / or national-level end-user's needs, workflow and decision-making process, both day-to-day and in the instance of a public safety emergency.
Impact. Submission details how the system will measurably advance current city-level and / or national-level practices, as well as how the system will integrate with and complement existing systems and technologies.
Feasibility. Submission demonstrates a reasonable path for implementation, and a clear method for validating data-driven signals with a high degree of confidence, backed by credible supporting evidence.
Sustainability. Extent to which the submission illustrates a plan for DHS to maintain consistent long-term access to the system, and adequately addresses potential constraints and potential unintended consequences of the system.
Scalability. Offers a plan as to how the system will expand to other geographic areas, to different signal types indicative of biothreat incidents, or to other scenarios of concern for public safety.
Team. Demonstrates commitment and an ability to further advance the system design into a prototype and / or operable system, and also demonstrates significant evolution and improvement of the initial concept over the course of the Virtual Accelerator.

Whitepapers and system designs were required as discrete assets to encourage a comprehensive plan for both data and operational viability. And the criteria, submission requirements, and design of the Virtual Accelerator for Stage 2 all were designed to encourage finalists to exhibit how they developed systems with a balanced focus on data, technology, and users.

The challenge's streamlined submission and evaluation process opened the doors to a wide array of expert solvers representing many disciplines beyond biodefense who can contribute their expertise and technology to the field. In fact, many judges remarked on how impressed they were by the quality of submissions and diversity of submitters.

In addition to the 37 teams that entered, many other potential solvers showed early interest by visiting the challenge website, registering for webinars and registering on Luminary Lightbox™. All of these audiences could be targeted for future efforts in this space.

Area of Excellence #3: "Execute the Communications Plan"

The communications and engagement plan for this challenge took into account a variety of audiences including solvers, stakeholders, and news media. While the stats provided below show a significant media footprint, more importantly this challenge opened new channels for direct communication and collaboration with cities across the country.

The challenge team invited representatives from cities across the country to collaborate in shaping this effort, through invitations to participate in a "Meeting of the Minds," serve on the judging or review panel, or join the Virtual Accelerator as a subject matter expert or mentor.

Cities joined the product development journey, providing user feedback and testing opportunities. Certain finalist teams stood out for their emphasis on prototyping with cities, collaborating directly with officials on research and preliminary user testing:

One Health Alert System: This team worked directly with the Cabarrus County Health Alliance to redesign the Daily Disease Report to produce more real-time results. They incorporated learnings from this process in their storyboarding exercise.
Pre-Syndromic Surveillance: This team formed a data use agreement with the New York City Department of Health and Mental Hygiene and formed a partnership with the New York City Department of Sanitation for early piloting. They used these partnerships for user testing and exhibited their workflows in their system design asset.

Throughout the process, city representatives recognized the relevance and expansion potential for these innovations and appreciated the broader applicability of the technology this challenge explored. As such, they welcomed the opportunity for future collaboration.

During the open submission period in Stage 1, the challenge team sought to drive submissions from innovators and communicate the program's value to the public and stakeholders. This was accomplished through a public press announcement and targeted outreach, solver and influencer activation, and sharing messages through the challenge (e.g., blog and newsletter) and various DHS S&T channels.

Press coverage: The challenge was released to hundreds of media outlets. DHS S&T confirmed stories in 10 outlets, including security (American Security Today), government (FedScoop), and healthcare (Fierce Healthcare).
Targeted outreach: Directly contacted 300+ validators and solvers; secured placement in Challenge.gov, NYC Open Data, Open Data Atlanta, Open Data D.C., and Harvard Business School Digital Initiative newsletter.
Challenge newsletter: Launch announcement was opened 200+ times via forwarding.
Social media: 39 tweets using the #HiddenSignals, generating 202,832 impressions. Notable tweets include: U.S. Chief Data Scientist DJ Patil (@DPatil), In-Q-Tel's B. Next Lab (@HarvardCIL), and data influencer (@KDNuggets).

These efforts resulted in 139 Informational Webinar RSVPs and 37 submissions.

During the winner announcement, the challenge team sought to drive awareness of the program to the public and stakeholders such as city-level employees, healthcare professionals, and data technology influencers across the United States. Promotion included a public press announcement and targeted outreach, influencer activation, and sharing messages through the challenge, DHS S&T, and challenge winners' channels. These efforts achieved several results during the winner announcement period of May 30, 2018 through June 7, 2018:

Press coverage: The winner press release was sent to hundreds of media outlets and to local media where the winners resided. Stories in five outlets: FedScoop, American Security Today, Homeland Security News, News Medical Life Sciences, and Homeland Prep News. The challenge also attracted interest from Wall Street Journal, Boston Globe, and GeekWire.
Social media: 69 tweets using #HiddenSignals, generating 624,000+ impressions. Tweets included data science influencer KDNuggets, challenge winner John Brownstein, HealthMap, Boston Health News Blog, Mallory Nobles, Virtual Accelerator speaker Kim Lucas, and bioinformatics news aggregator Bio Systs.
Targeted outreach: Directly contacted more than 50 experts involved with the challenge, some of whom also spread the word throughout their networks.
Challenge newsletter: The winner announcement newsletter was opened more than 225 times via forwarding, including opens by Cabarrus Health, Human Patterns, and Harvard Business School.
Challenge winner outlets: The challenge winners amplified the news through social media and blogs for the New York University Center for Urban Science + Progress and Carnegie Mellon University Heinz College.

These efforts resulted in nearly 300 visitors to the challenge website over three days, including:

Tech giants Amazon and Microsoft.
City-level employees from New York City Transit Authority and City of Palo Alto.
Hospitals such as Longwood Medical and Academic Area, Children's Hospital Colorado, London School of Hygiene and Tropical Medicine, and St. Jude's Children's Hospital.

Challenge Type: Analytics, Ideas, Software, Technology

DHS S&T decided to address gaps in biothreat identification through an open innovation challenge to engage a diverse community of solvers and access new thinking; maximize available traditional and non-traditional data resources; and accelerate concept development process to save lives faster.

The solutions sought and received bridged several types, including analytics, ideas, software, and technology. The challenge specifically focused on ideas in these solution areas:

Producing real-time alerts by prioritizing fast-acting biothreats and using sustainable real-time data streams
Providing actionable insights that enable decision makers to launch interventions (or choose not to intervene) with confidence
Detecting abnormal conditions (rather than just specific pathogens), because in some cases general awareness of a biothreat can be enough to intervene and knowledge of the specific pathogen isn't required for an effective early warning

By starting with concepts and then allowing finalists to further develop those concepts into actual system designs, the challenge set up a natural bridge between ideas and actionable solutions that will carry on past the end of the challenge. As such, it created buy-in and enthusiasm for future collaboration among all those involved, including solvers, subject matter experts, and stakeholders.

Legal Authority

America COMPETES Act