Pediatric COVID-19 Data Challenge
Help develop computational models to predict children at risk for severe COVID-19 complications
Department of Health and Human Services - Biomedical Advanced Research and Development Authority (BARDA)
Type of Challenge: Software and apps, Analytics, visualizations, algorithms, Scientific
Partner Agencies | Federal: National Center for Advancing Translational Sciences (NCATS), Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), Health Resources & Services Administration (HRSA) Maternal & Child Health Bureau
Partner Agencies | Non-federal: Sage Bionetworks
Submission Start: 08/19/2021 12:00 AM ET
Submission End: 11/19/2021 11:59 PM ET
While most children with COVID-19 are asymptomatic or have mild symptoms, healthcare providers have difficulty determining which of their pediatric patients will progress to moderate or severe COVID-19 early in the progression. Some of these patients develop multisystem inflammatory syndrome in children (MIS-C), a life-threatening inflammation of organs and tissues. Methods to distinguish children at risk for severe COVID-19 complications, including conditions such as MIS-C are needed for earlier interventions to improve pediatric patient outcomes.
Multiple HHS divisions are coming together for a data challenge competition that will leverage de-identified electronic health record data to develop, train and validate computational models that can predict severe COVID-19 complications in children, equipping healthcare providers with the information and tools they need to identify pediatric patients at risk.
The challenge is sponsored by the Biomedical Advanced Research and Development Authority (BARDA), part of the HHS Office of the Assistant Secretary for Preparedness and Response, in partnership with two institutes from the National Institutes of Health – the National Institute of Health’s National Center for Advancing Translational Sciences (NCATS) and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) – along with the Health Resources and Services Administration’s (HRSA) Maternal and Child Health Bureau. The challenge will be administered by Sage Bionetworks.
Challenge participants will be asked to develop, train, and validate computational models to predict pediatric patients at risk for hospitalization, need for ventilation, and cardiovascular interventions, utilizing de-identified electronic health record data available through NCATS’ National COVID Cohort Collaborative (N3C) Data Enclave. The N3C Data Enclave is a central, harmonized data repository that represents electronic health records from over 55 health centers across the U.S. To protect patient privacy, de-identified data provides information useful to researchers without revealing any information that could identify individual patients.
BARDA is part of the office of the Assistant Secretary for Preparedness and Response at the U.S. Department of Health and Human Services.
- Aug. 19 - Onboarding Phase
- Sep. 15 - Round 1 Begins
- Oct. 12 - Round 2 Begins
- Nov. 19 - End Round 1 & 2
- TBD - Winners Announced
- TBD - Collaboration Phase Begins
Total Cash Prize Pool
- $200,000 split among up to three winners
- Prize distribution will occur at the sole discretion of BARDA.
Non-monetary Prizes Winners may also be eligible to apply for follow on funding support and in-kind services from BARDA for further technology development and clinical evaluation.
Eligibility Rules for Participating in the Competition: To be eligible to win a prize under this Challenge, an individual or entity—
- Must have registered to participate in the competition under the rules promulgated by HHS
- Must have complied with all the requirements under this section
- In the case of a private entity, must be incorporated in and maintain a primary place of business in the United States, and in the case of an individual, whether participating singly or in a group, must be a citizen or permanent resident of the United States
- May not be a federal entity or federal employee acting within the scope of their employment (all non-HHS federal employees must consult with their agency Ethics Official to determine whether the federal ethics rules will limit or prohibit the acceptance of a COMPETES Act prize);
- Must not be an HHS employee
- Federal grantees may not use federal funds to develop submissions unless consistent with the purpose of their grant award; and
- Federal contractors may not use federal funds from a contract to develop COMPETES Act challenge applications or to fund efforts in support of a COMPETES Act challenge submission.
- An individual or entity must not be deemed ineligible because the individual or entity used federal facilities or consulted with federal employees during a competition if the facilities and employees are made available to all individuals and entities participating in the competition on an equitable basis.
- Participants must also agree to assume any and all risks and waive claims against the federal government, technical reviewers, challenge administrators and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from participation in this prize contest, whether the injury, death, damage, or loss arises through negligence or otherwise.
- Participants are not required to obtain liability insurance or demonstrate financial responsibility in any specified amount for claims by a third party for death, bodily injury, or property damage or loss resulting from an activity carried out in connection with participation in a prize competition, but participants are encouraged to consult with their advisors with respect to the level of insurance that is prudent for each participant.
- Registered participants agree to indemnify the Federal Government against third-party claims for damages arising from or related to prize competition activities, and further indemnify the federal government for damage or loss to government property resulting from such activities.
- The U.S. Department of Health and Human Services’ Office of the Assistant Secretary for Preparedness and Response (HHS/ASPR) reserves the right to cancel, suspend, and/or modify the challenge, or any part of it, for any reason, at HHS/ASPR’s sole discretion.
Intellectual Property (IP) Rights
- Participants should be aware that any agreement signed or obligation undertaken in regard to their participation in this prize competition that conflicts with the prize competition rules, terms, and conditions may result in disqualification of the participant’s submission.
- By participating in this prize competition, each participant (whether participating singly or in a group) warrants that he or she is the sole author or owner of, or has the right to use, any copyrightable works that the submission comprises, that the works are wholly original with the participant (or are an improved version of an existing work that the participant has sufficient rights to use and improve), and that the submission does not infringe any copyright or any other rights of any third party of which the participant is aware. Each participant will retain all other intellectual property rights in their submissions, as applicable. To participate in the prize competition, each participant must warrant that there are no legal obstacles to providing the above-referenced nonexclusive licenses of participant rights to the federal government. To receive an award, winners will retain ownership of their intellectual property rights in the solution but must grant to the federal government the nonexclusive, nontransferable, irrevocable, paid-up license to practice, or have practiced for or on its behalf, the solution throughout the world for federal purposes.
- Each participant must clearly delineate any intellectual property (IP) and/or confidential commercial information contained in a submission that the participant wishes to protect as proprietary data, code, and algorithms. By participating in this prize challenge competition, challenge participants understand that their submission may be reviewed by a panel of federal and non-federal technical reviewers.
- All materials submitted to HHS as part of a submission become HHS agency records. Any proprietary information contained in a submission must be clearly designated at the time of submission.
- If the entry includes any third-party works (such as third-party content or open source code), the participant must be able to provide, upon request, documentation of all appropriate licenses and releases for use of such third-party works. If the participant cannot provide documentation of all required licenses and releases, HHS reserves the right, at their sole discretion, to disqualify the entry.
Additional rules are outlined here: https://www.synapse.org/peds_covid_severity
Technical reviewers with expertise relevant to the Challenge will evaluate the solutions based on their ability to achieve the criteria listed below. Technical reviewers will be comprised of experts sourced from the Federal government, academia, industry, and other sectors as deemed relevant. The solutions and evaluation information from the technical reviewers will then be reviewed by federal employees serving as judges, who will select up to 3 challenge winners, as well as any honorable mentions subject to the final decision by the Award Approving Official.
The award approving official will be the BARDA director.
Each submission will be evaluated along quantitative and qualitative metrics.
Participants in this challenge will build models on a training dataset established by challenge organizers. Those trained models will then be tested on a holdout set to establish initial model performance. Models will then be put through a “final training”. The primary metrics used to score models will be the Fβ metric, followed by the Area Under the Precision Recall Curve and the Area Under the Receiver Operator Curve. The AUROC will be used to evaluate model generalizability, where models will be evaluated for temporal, cross-site, and demographic generalizability.
Qualitative metrics of utility and reproducibility will also be used as part of the final ranking of the submissions. Factors related to model utility include (1) feature interpretation (Can the factors driving individual predictions be understood or is the model a black box?), (2) method clarity (Are the code, documentation, and final submission write up understandable and clear?), (3) timeliness of predictions (are models able to make accurate predictions before an intervention needs to happen), and (4) context utility (can a model still make accurate predictions in the setting in which it will be used?). Factors related to reproducibility include (1) technical reproducibility (Can the code be deployed and re-run successfully under different computational conditions?), (2) prediction reproducibility (Does the model make the same predictions each time under the same conditions?), and (3) documentation reproducibility (Is there sufficient documentation to replicate the model?).
The above metrics will become part of a ranking consensus for the submissions in each of the challenge questions. The primary purpose of this challenge is to identify the most clinically useful predictions models for pediatric severity of COVID-19. The judges will use both the quantitative metrics and the qualitative metrics to identify the models submitted to each question that are most likely to be clinically useful.
For more information: https://www.synapse.org/peds_covid_severity
How to Enter
Registration with the challenge
- Step 1: Create an account on Synapse.org. Make sure your account information (name and institution) match the information in your N3C Enclave account.
- Step 2: Register for the challenge using the registration button.
- Step 3: Create and register a team (or join an existing team)
Onboarding into the N3C Enclave
- Step 1: Check if your institution has a data use agreement (DUA) in place with the N3C organization. (https://covid.cd2h.org/duas) If not, have your institution sign a DUA with N3C. (https://ncats.nih.gov/n3c/about/applying-for-access)
- Step 2: Clarify if you need Institutional Review Board (IRB) approval to use de-identified clinical data for research per your institution’s research policies.
- Step 3: Register with N3C. (https://labs.cd2h.org/registration/) Obtain an N3C Data Enclave account, complete the “Information Security, Counterintelligence, Privacy Awareness, Records Management Refresher, Emergency Preparedness Refresher” course, and complete home institution’s human subjects research training to access de-identified data (https://ncats.nih.gov/n3c/about/applying-for-access)
- Step 4: Request access to the project and agree to N3C Data User Code of Conduct and Data Use Request project description for the challenge project space in the enclave.
For more information and instructions: https://www.synapse.org/peds_covid_severity
Point of Contact
Have feedback or questions about this challenge? Send the challenge manager an email