Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Informational Only

This challenge is no longer accepting new submissions.

Army Signal Classification Challenge

The Army is seeking innovative approaches, leveraging machine learning/deep learning, to conduct blind radio frequency signal analysis.

Department of Defense

Total Cash Prizes Offered: $150000
Type of Challenge: Scientific
Partner Agencies | Federal: 1
Submission Start: 04/30/2018 12:00 AM ET
Submission End: 08/13/2018 12:00 AM ET

This challenge is externally hosted.

You can view the challenge details here: https://sites.mitre.org/armychallenge

Description

Registration will be conducted on a third-party site: https://sites.mitre.org/armychallenge/ Did you know the Army Rapid Capabilities Office (RCO) is sponsoring a Blind Signal Classification Challenge where the winners will split $150,000? Set to launch on April 30, the challenge is seeking new technologies that apply artificial intelligence and machine learning to signal identification and classification. This effort invites anyone who can bring machine learning algorithms and the supporting processes, methods and tools needed to improve the speed and agility of blind signal identification and classification within the electromagnetic spectrum. The RCO hopes the technology from the challenge will lead to advancements in how Electronic Warfare Officers identify and react to these signals on the battlefield. The Government intent is for solvers to present advanced algorithms and AI implementations with a high degree of classification accuracy and performance (e.g., speed of classification, low CPU resource requirement) that would allow for ease of integration within existing systems.  This research goal is to promote innovation and advancement in the area of signal processing. In exchange for cash prizes, the government is seeking a brief technical paper describing the methodology of the specific implementation, to include both the model architecture and training process, including the hyper-parameter optimization process. In order to fully understand the approach, essential source code elements must be provided that show the implementation of the model and training process. Participants’ solutions for each test dataset will be scored independently based on a cross-entropy loss function. Participants will have an aggregate score based on their individual CSV file submissions for each test dataset. Upon registration and approval participants will be given access to a third-party site to obtain access to the training dataset.

Judging Criteria

Multi-Class Logarithmic Loss Metric Scoring Metric Submissions are evaluated using a multi-class logarithmic loss metric, with scores ranging from 0 to 100. Each test instance has been labeled with one true class of 24 possible classes. For each test instance, i, you must submit a set of 24 predicted probabilities, pi,j., where j=1, 2, …, 24. The formula used to compute your score is then,

  Submission File Participants will submit CSV files that contain their results across the test instances. These results are expected in the format of a confusion matrix that shows the predicted probabilities across each of the modulation classes for each test instance. Results will be scored from a scale of 0-100, with 100 being the highest possible score. The logloss function will be calculated across the submitted results and the resulting score will be a function of 100/(1-logloss). Participant solutions must be submitted in CSV format for each test dataset, and must provide predicted probabilities for every test instance and modulation class. An example CSV file is provided on the challenge site for competitor reference. Your submitted file for test set 1 must be named “TestSet1Predictions.csv” and your predictions for test set 2 must be named “TestSet2Predictions.csv”. The 24 modulations to predict are: BPSK,QPSK, 8PSK, 16PSK, QAM16, QAM64, 2FSK_5KHz, 2FSK_75KHz, GFSK_75KHz, GFSK_5KHz, GMSK, MSK, CPFSK_75KHz, CPFSK_5KHz, APSK16_c34, APSK32_c34, QAM32, OQPSK, PI4QPSK, FM_NB, FM_WB, AM_DSB, AM_SSB, NOISE The submission format is ID,BPSK,QPSK, 8PSK, 16PSK, QAM16, QAM64, 2FSK_5KHz, 2FSK_75KHz, GFSK_75KHz, GFSK_5KHz, GMSK, MSK, CPFSK_75KHz, CPFSK_5KHz, APSK16_c34, APSK32_c34, QAM32, OQPSK, PI4QPSK, FM_NB, FM_WB, AM_DSB, AM_SSB, NOISE 1,0.0123,0.0234,...,0.0345 2,0.03,0.01,0.06,...,0.4 ... with the first (header) row giving the test instance “ID” and the 24 modulation classes. Note: In addition to the initial labeled training dataset provided upon the competition open date, solvers will be provided the first unlabeled test dataset approximately sixty (67) days after the start of the competition, with unlabeled test set 2 provided approximately fifteen (15) days after providing test set 1.  All participants must include results for both test datasets to be considered for award.

Prizes

Signal Classification Solver Awards
Cash Prize Amount: $150000
Solver with winning algorithms who meet government terms and conditions will receive monetary awards from the government.

Rules

Participants must register on the website https://sites.mitre.org/armychallenge/to enter. All submissions must conform to the requirements set forth on https://sites.mitre.org/armychallenge/.