follow us on facebook follow us on twitter email us

Army Signal Classification Challenge

About the Challenge
The Army is seeking innovative approaches, leveraging machine learning/deep learning, to conduct blind radio frequency signal analysis.

Posted By: Department of Defense--Military Programs, Research, Development, Test, and Evaluation
Category: Scientific/Engineering
Skill: Algorithms Interest: Science & Research Partnership With: Department of Defense
Submission Dates: 12 a.m. ET, Apr 30, 2018 - 12 a.m. ET, Jul 30, 2018 Judging Dates: Jul 06, 2018 - Jul 30, 2018 Winners Announced: Aug 13, 2018

Registration will be conducted on a third-party site: https://sites.mitre.org/armychallenge/

Did you know the Army Rapid Capabilities Office (RCO) is sponsoring a Blind Signal Classification Challenge where the winners will split $150,000? Set to launch on April 30, the challenge is seeking new technologies that apply artificial intelligence and machine learning to signal identification and classification. This effort invites anyone who can bring machine learning algorithms and the supporting processes, methods and tools needed to improve the speed and agility of blind signal identification and classification within the electromagnetic spectrum. The RCO hopes the technology from the challenge will lead to advancements in how Electronic Warfare Officers identify and react to these signals on the battlefield.

The Government intent is for solvers to present advanced algorithms and AI implementations with a high degree of classification accuracy and performance (e.g., speed of classification, low CPU resource requirement) that would allow for ease of integration within existing systems.  This research goal is to promote innovation and advancement in the area of signal processing. In exchange for cash prizes, the government is seeking a brief technical paper describing the methodology of the specific implementation, to include both the model architecture and training process, including the hyper-parameter optimization process. In order to fully understand the approach, essential source code elements must be provided that show the implementation of the model and training process. Participants’ solutions for each test dataset will be scored independently based on a cross-entropy loss function. Participants will have an aggregate score based on their individual CSV file submissions for each test dataset.

Upon registration and approval participants will be given access to a third-party site to obtain access to the training dataset.


Judging Criteria

Multi-Class Logarithmic Loss Metric

Scoring Metric

Submissions are evaluated using a multi-class logarithmic loss metric, with scores ranging from 0 to 100. Each test instance has been labeled with one true class of 24 possible classes. For each test instance, i, you must submit a set of 24 predicted probabilities, pi,j., where j=1, 2, …, 24. The formula used to compute your score is then,

 

Submission File

Participants will submit CSV files that contain their results across the test instances. These results are expected in the format of a confusion matrix that shows the predicted probabilities across each of the modulation classes for each test instance. Results will be scored from a scale of 0-100, with 100 being the highest possible score. The logloss function will be calculated across the submitted results and the resulting score will be a function of 100/(1-logloss).

Participant solutions must be submitted in CSV format for each test dataset, and must provide predicted probabilities for every test instance and modulation class. An example CSV file is provided on the challenge site for competitor reference.

Your submitted file for test set 1 must be named “TestSet1Predictions.csv” and your predictions for test set 2 must be named “TestSet2Predictions.csv”.

The 24 modulations to predict are: BPSK,QPSK, 8PSK, 16PSK, QAM16, QAM64, 2FSK_5KHz, 2FSK_75KHz, GFSK_75KHz, GFSK_5KHz, GMSK, MSK, CPFSK_75KHz, CPFSK_5KHz, APSK16_c34, APSK32_c34, QAM32, OQPSK, PI4QPSK, FM_NB, FM_WB, AM_DSB, AM_SSB, NOISE

The submission format is

ID,BPSK,QPSK, 8PSK, 16PSK, QAM16, QAM64, 2FSK_5KHz, 2FSK_75KHz, GFSK_75KHz, GFSK_5KHz, GMSK, MSK, CPFSK_75KHz, CPFSK_5KHz, APSK16_c34, APSK32_c34, QAM32, OQPSK, PI4QPSK, FM_NB, FM_WB, AM_DSB, AM_SSB, NOISE
1,0.0123,0.0234,…,0.0345
2,0.03,0.01,0.06,…,0.4

with the first (header) row giving the test instance “ID” and the 24 modulation classes.

Note: In addition to the initial labeled training dataset provided upon the competition open date, solvers will be provided the first unlabeled test dataset approximately sixty (67) days after the start of the competition, with unlabeled test set 2 provided approximately fifteen (15) days after providing test set 1.  All participants must include results for both test datasets to be considered for award.

Prizes
Signal Classification Solver Awards $150,000.00 Solver with winning algorithms who meet government terms and conditions will receive monetary awards from the government.
Solutions
No solutions have been posted for this challenge yet.
Rules

Participants must register on the website https://sites.mitre.org/armychallenge/to enter.  All submissions must conform to the requirements set forth on https://sites.mitre.org/armychallenge/. 

 

Submit Solution
Submissions for this competition are being accepted on a third-party site. Please visit the external site for instructions on submitting: https://sites.mitre.org/armychallenge/
Challenge Followers
Public Profile: 0
Private Profile: 4