follow us on facebook follow us on twitter email us

AHRQ Step Up App Challenge

About the Challenge
Advancing Care Through Patient Self-Assessments

Posted By: Agency for Healthcare Research and Quality
Category: Software/Apps
Skill: Software/Apps Interest: Health Submission Dates: 9 a.m. ET, Aug 13, 2018 - 11:59 p.m. ET, Sep 24, 2018 Judging Dates: Sep 24, 2018 - Oct 24, 2018 Winners Announced: Oct 26, 2018

**Please email your submission to after 4pm ET on September 24**

Video Message From AHRQ Director Gopal Khanna About Entering the Step Up App Challenge

Unable to attend the Step Up App Challenge: Phase 1 Informational Webinar? Check out the slides presented during the August 23 webinar

The Background

Patient-reported outcomes (PROs) are vital to understanding the complex needs of a patient. For this challenge, PRO data are defined as “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or other medical expert.” In other words, PRO data come from patient self-assessments and look beyond the clinical data to assess how a patient feels about his or her health.

The Problem

While PRO data have proven useful to healthcare providers, they are not widely used in clinical settings. Recently, some digital tools have been developed to streamline the collection of PRO data. However, those digital tools are not widely adopted due to challenges related to clinician workflow and patient ease with the tools.  Also, researchers who want to analyze PRO data across different practices or health systems often encounter the issue that data are not collected in a standardized manner. Creating a digital tool to efficiently collect, aggregate, and share PRO data is critical to advancing the quality of care.

Act Now

The Agency for Healthcare Research and Quality (AHRQ) Step Up App Challenge is seeking innovative technologists to develop user-friendly apps that collect standardized PRO data for clinical and research purposes.

Submit your app idea by September 24, 2018 for a chance to win the grand prize of up to $87,000! Register for the challenge today.

Frequently Asked Questions 

Submission Requirements

Overview of Phase 1 – Proposals

The proposal phase allows participants to present plans and frameworks that demonstrate how they plan to carry out the application development. In this phase, participants should describe the technical, operational, and financial aspects of their proposed approaches. The main goal of Phase 1 is for participants to show the originality, feasibility, and executability of their application development approaches.

Phase 1 Submission Requirements:

  • Applications should be in English and all documents uploaded in PDF format
  • Submit by the deadline of September 24, 2018 using the online platform:
  • The submission should be made up of, minimally, a title and two uploaded .pdf files.  The two files are described in more detail below.  You may also optionally include a picture/logo (for instance a logo), and a company’s URL
  • Business case (5-page maximum)
    • Include an executive summary stating the value proposition:
      • Describe the specific problem being solved
      • Analysis of the existing market and market opportunity
      • Describe how the proposed application will solve the problem
      • Describe the customers/users
    • Describe the methods and technologies proposed to develop the application
    • Describe the business model:
      • Describe the specific problem being solved
      • Analysis of the existing market and market opportunity
      • Describe how the proposed application will solve the problem
      • Describe the customers/users
    • Development plan and timeline
      • Describe key activities and resources required to build the application
      • Plan to make the application readily available to patients. For example, to be used on existing mobile platforms or deployed on a public-facing website
    • Define metrics for success (e.g., number of users of the application, money saved by using the application, time saved, increases in patient-reported outcome data exchanges between patients and providers)
    • Show potential risks and mitigation strategies, including security constraints
    • Describe participant roles, responsibilities, and capabilities. A multi-disciplinary team is encouraged.
  • Include a brief deck presentation PDF of the application and use case(s) to visually present the project (10 slides maximum)
    • Content:
      • Briefly describe the proposed approach and how the participants will develop an application using the FHIR technical specifications and PROMIS® physical functioning measures provided by AHRQ
      • Explain the competitive advantage of the approach:
        • Innovative and does not infringe on in-market solutions
        • Includes an advanced application technology that has the potential to improve the collection and use of PRO data
        • Designed for patient engagement
      • Give an example use case
      • Show proposed development workflow and deliverables
      • An architecture diagram or framework
  • The above documents should discuss how you will comply with the Health Insurance Portability and Accountability Act (HIPAA) if applicable.

Overview of Phase 2 – Development

Phase 1 winners will advance to Phase 2 focused on developing the application using the FHIR technical specifications and PROMIS® resources provided by AHRQ. The application should have the capability to do CATs which requires the device to be connected with a server for real-time question calling and scoring (online administration). Participants can use the Assessment Center Application Programming Interface (API) to build an application that enables CAT. In addition to CAT, participants are highly encouraged to also include the capability to administer the selected PROMIS® short form v1.0 – Physical Function 10a (offline administration). Below are the references to PROMIS® API, CAT, and selected short form. A webinar on how to use the PROMIS® API will be held at the beginning of Phase 2.

Register for credentials to use the API:

Documentation on how to administer PROMIS® items using the API:

The goal of Phase 2 is to present a viable application to collect standardized PRO data in ambulatory care settings including both primary care and specialty care practices. Participants need to demonstrate that the app could export standardized data which would facilitate future data integration into an EHR or IT system. Participants are required to undertake user testing of the application during Phase 2. Participants must provide evidence of testing with at least five people. A larger amount of time spent with each tester, greater formal rigor, and the number and diversity of people used for testing will result in a more positive assessment under the judging criteria. Evidence demonstrating user testing could include sample feedback, quotes, or pictures, and should include how it affected the development of the application.

 Phase 2 Submission Requirements

  • Submit by the deadline of February 15, 2019
  • Submit the application in a PDF format
  • Provide an implementation plan (up to 10 pages)
    • Describe key activities and resources required to deploy and pilot the application
    • Refine and incorporate relevant information from business case submission in Phase 1
  • Provide evidence of user testing including how it affected the application development (up to 5 pages)
  • Apps must incorporate the following usability, functionality, and technical requirements:
    • Computerized adaptive testing (CAT)
    • Adaptable to PRO domains other than physical functioning (e.g., pain, mental health)
    • Simple design, easy navigation, and intuitive to use
    • Personalized content
    • Not connected with any social media applications
    • HIPAA compliance if applicable
    • Device-based security
      • The PRO data must be encrypted at rest, even when the device is unlocked.
      • Follow specifications and current recommendations for AES-256 bit encryption
      • The key must be saved in the Keychain or SharedPreferences for iOS and Android respectively.
    • Authentication security
      • The application must be able to support token-based authentication including the standards of SAML and OAuth2.
      • The application must be capable of completing a onetime registration process with an external system leveraging two-factor authentication by creating an account with username and password, and a pin sent to the phone number provided during registration.
    • Data communication
      • Demonstrate the ability to represent the collected PRO data as both JavaScript Object Notation (JSON) and Extensible Markup Language (XML)
  • Technical ability to do video demonstration of the application via live webinar to display the application’s device and data security, usability, accessibility, ease of data collection and HIPAA compliance if applicable
  • Export and deposit standardized PRO data to a FTP site selected by AHRQ

Overview of Phase 3 – Pilot Test

The final phase will involve pilot testing the application in “real-life” clinical settings. The grand prize winner in Phase 2 is required to collaborate with MedStar Health to pilot test the application in nine practice settings in the District/Maryland/Virginia area. If the grand prize winner cannot be at the pilot sites physically, MedStar could facilitate remote access for the collaboration. There will be a mix of primary care and specialty care practices. The goal of Phase 3 is to assess the usability of the application. This phase will also test the feasibility of implementation, the scalability of the application, and acceptance by stakeholders.

 Phase 3 Requirements

  • Work with MedStar Health on the pilot test including setting up the IT infrastructure, modifying the application if needed, and troubleshooting issues during data collection
    • Pilot Test Period: Monday, February 25, 2019 – Monday, September 9, 2019
  • Demonstrate achievement of milestones to receive the final prize payment
    • Post an open source version of the application’s code on the GitHub source repository to make the code publicly available
    • Successful data collection from all pilot sites
    • Successful export of standardized data at all pilot sites
  • Submit monthly progress report of the pilot test (25th of every month from Feb – Sept, 2019)
  • Submit a final report by September 13, 2019 on lessons learned, recommendations, and concrete next steps for sustainability including commercialization and/or broadened use and how to attract patients and/or providers to adopt and use the application

Basis Upon Which Winner Will be Selected

Phase 1 Evaluation Criteria:

  • Team/Participant Capabilities (20%)
    • There is appropriate expertise and capability to bring the idea to the development stage. A multi-disciplinary team is encouraged.
    • The team and/or participant has the resources available to carry out the proposed work
  • Impact (30%)
    • The proposed Solution has the potential to improve the state of PRO collection, aggregation, and reporting
    • The proposed Solution describes the value proposition to users
    • There is a clear plan to make the proposed Solution readily available to patients and providers on existing mobile platforms, or customer-facing websites
  • Feasibility (30%)
    • The proposed Solution demonstrate a reasonable and credible approach to accomplish the proposed objectives, tasks, outcomes, and deliverables
    • The proposed Solution addresses a reasonable pathway or timeline to application development
    • The proposed Solution clearly defines potential risks and mitigation strategies including security constraints
    • The proposed Solution includes an architecture diagram or framework that is feasible and executable for deployment
  • Originality (20%)
    • The proposed Solution is innovative and does not infringe on in-market solutions
    • The proposed Solution includes an advanced application technology that has the potential to improve the collection and use of PRO data
    • The proposed solution is designed for patient engagement

Phase 2 Evaluation Criteria:

  • Technical Merit (40%)
    • The application demonstrates technical compliance regarding
      • Device-based security
      • Authentication security
      • Data communication
      • FHIR standards provided by AHRQ
      • Export and deposit standardized PRO data to a FTP site selected by AHRQ
    • The application’s technical methods and frameworks are sound and appropriate for the given problem and user base
    • The solution utilizes best-practices and draft FHIR technical specifications in the development of the application
  • Usability and functionality (30%)
    • The application has a simple design, and is intuitive to use and easy to navigate
    • The application enables users to easily collect, aggregate, and manage PRO data
    • The application can administer computerized adaptive test (CAT)
    • The application is not connected with any social media applications
    • The application was tested with five users and is designed with an understanding of the needs and goals of providers and patients
    • The application has the capability to adapt and embed a variety of PRO measures other than physical functioning
  • Deployability (30%)
    • The application can be easily adopted by a healthcare practice (e.g., use open and flexible industry standards)
    • The team addresses all risks and mitigation strategies faced in a real-life pilot test
    • The application has a mechanism to capture and leverage user data/feedback in the pilot test
    • The application has the capacity to scale to multiple healthcare practices
    • The application adequately describes sustainability plans beyond the completion of this Challenge
How to Enter

Participants can participate by visiting and creating a account. Once logged in, click “Submit Solution” anytime during the submission period stated above. Submission requirements and Challenge information including the judging criteria will be provided at

Phase 1 $120,000.00 Up to 10 winners receive up to $12,000 each
Phase 2 $90,000.00 1st place winner receives up to $35,000 2nd place winner receives up to $30,000 3rd place winner receives up to $25,000
Phase 3 $40,000.00 Grand Prize Winner receives $40,000 after successful pilot test

Add to the Discussion


Eligibility Rules for Participating in the Challenge

To be eligible to win a prize under this Challenge, an individual or entity:

  1. Shall have registered to participate in the Challenge under the rules promulgated by the Agency for Healthcare Research and Quality.
  2. Shall have complied with all the stated requirements of the Step Up App Challenge
  3. In the case of a private entity, shall be incorporated in and maintain a primary place of business in the United States, and in the case of an individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States.
  4. May not be a Federal entity or Federal employee acting within the scope of their employment.
  5. Shall not be an HHS employee working on their applications or Submissions during assigned duty hours.
  6. Shall not be an employee of the Agency for Healthcare Research and Quality.
  7. Federal grantees may not use Federal funds to develop COMPETES Act Challenge applications unless consistent with the purpose of their grant award.
  8. Federal contractors may not use Federal funds from a contract to develop COMPETES Act Challenge applications or to fund efforts in support of a COMPETES Act Challenge Submission.
  9. A product may be disqualified if it fails to function as expressed in the description provided by the Participant, or if it provides inaccurate or incomplete information.
  10. If applicable, the proposed application must be HIPAA compliant to be eligible for entry into the Challenge.
  11. All individual members of a team must meet the eligibility requirements.
  12. Shall not be currently under contract with the Office of the National Coordinator for Health Information Technology (ONC) to perform work related to electronic capture of patient-reported outcomes or be a subcontractor under such contract.

An individual or entity shall not be deemed ineligible because the individual or entity used Federal facilities or consulted with Federal employees during a Challenge if the facilities and employees are made available to all individuals and entities participating in the Challenge on an equitable basis.

By participating in this Challenge, Participants:

  1. Agree to assume any and all risks and waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from their participation in this prize contest, whether the injury, death, damage, or loss arises through negligence or otherwise.
  2. Agree to obtain liability insurance, or demonstrate financial responsibility, in the amount of $500,000, for claims by (a) a third party for death, bodily injury, or property damage, or loss resulting from an activity carried out in connection with participation in a Challenge, with the Federal Government named as an additional insured under the registered participant’s insurance policy, and (b) the Federal Government for damage or loss to Government property resulting from such activity.
  3. Agree to indemnify the Federal Government against third party claims for damages arising from or related to Challenge activities.

General Submission Requirements

In order for a Submission to be eligible to win this Challenge, it must meet the following requirements:

  1. No HHS or AHRQ logo—They must not use HHS’ or AHRQ’s logos or official seals and must not claim endorsement.
  2. Functionality/Accuracy—An application may be disqualified if it fails to function as expressed in the description provided by the participant, or if it provides inaccurate or incomplete information.
  3. Security—Submissions must be free of malware. Participant agrees that AHRQ may conduct testing on the application(s) to determine whether malware or other security threats may be present. AHRQ may disqualify the application(s) if, in AHRQ’s judgment, the application may damage government or others’ equipment or operating environment.


Submit Solution
Challenge Followers
Public Profile: 0
Private Profile: 33