Posted By: Department of Defense--Military Programs, Research, Development, Test, and Evaluation
Category: Scientific/Engineering Skill: Algorithms Interest: Science & Research Partnership With: Department of Defense
Submission Dates: 12 a.m. ET, Apr 30, 2018 - 12 a.m. ET, Jul 30, 2018 Judging Dates: Jul 06, 2018 - Jul 30, 2018 Winners Announced: Aug 13, 2018
Registration will be conducted on a third-party site: https://sites.mitre.org/armychallenge/
Did you know the Army Rapid Capabilities Office (RCO) is sponsoring a Blind Signal Classification Challenge where the winners will split $150,000? Set to launch on April 30, the challenge is seeking new technologies that apply artificial intelligence and machine learning to signal identification and classification. This effort invites anyone who can bring machine learning algorithms and the supporting processes, methods and tools needed to improve the speed and agility of blind signal identification and classification within the electromagnetic spectrum. The RCO hopes the technology from the challenge will lead to advancements in how Electronic Warfare Officers identify and react to these signals on the battlefield.
The Government intent is for solvers to present advanced algorithms and AI implementations with a high degree of classification accuracy and performance (e.g., speed of classification, low CPU resource requirement) that would allow for ease of integration within existing systems. This research goal is to promote innovation and advancement in the area of signal processing. In exchange for cash prizes, the government is seeking a brief technical paper describing the methodology of the specific implementation, to include both the model architecture and training process, including the hyper-parameter optimization process. In order to fully understand the approach, essential source code elements must be provided that show the implementation of the model and training process. Participants’ solutions for each test dataset will be scored independently based on a cross-entropy loss function. Participants will have an aggregate score based on their individual CSV file submissions for each test dataset.
Upon registration and approval participants will be given access to a third-party site to obtain access to the training dataset.
Multi-Class Logarithmic Loss Metric
Submissions are evaluated using a multi-class logarithmic loss metric, with scores ranging from 0 to 100. Each test instance has been labeled with one true class of 24 possible classes. For each test instance, i, you must submit a set of 24 predicted probabilities, pi,j., where j=1, 2, …, 24. The formula used to compute your score is then,
Participants will submit CSV files that contain their results across the test instances. These results are expected in the format of a confusion matrix that shows the predicted probabilities across each of the modulation classes for each test instance. Results will be scored from a scale of 0-100, with 100 being the highest possible score. The logloss function will be calculated across the submitted results and the resulting score will be a function of 100/(1-logloss).
Participant solutions must be submitted in CSV format for each test dataset, and must provide predicted probabilities for every test instance and modulation class. An example CSV file is provided on the challenge site for competitor reference.
Your submitted file for test set 1 must be named “TestSet1Predictions.csv” and your predictions for test set 2 must be named “TestSet2Predictions.csv”.
The 24 modulations to predict are: BPSK,QPSK, 8PSK, 16PSK, QAM16, QAM64, 2FSK_5KHz, 2FSK_75KHz, GFSK_75KHz, GFSK_5KHz, GMSK, MSK, CPFSK_75KHz, CPFSK_5KHz, APSK16_c34, APSK32_c34, QAM32, OQPSK, PI4QPSK, FM_NB, FM_WB, AM_DSB, AM_SSB, NOISE
The submission format is
ID,BPSK,QPSK, 8PSK, 16PSK, QAM16, QAM64, 2FSK_5KHz, 2FSK_75KHz, GFSK_75KHz, GFSK_5KHz, GMSK, MSK, CPFSK_75KHz, CPFSK_5KHz, APSK16_c34, APSK32_c34, QAM32, OQPSK, PI4QPSK, FM_NB, FM_WB, AM_DSB, AM_SSB, NOISE
with the first (header) row giving the test instance “ID” and the 24 modulation classes.
Note: In addition to the initial labeled training dataset provided upon the competition open date, solvers will be provided the first unlabeled test dataset approximately sixty (67) days after the start of the competition, with unlabeled test set 2 provided approximately fifteen (15) days after providing test set 1. All participants must include results for both test datasets to be considered for award.
Participants must register on the website https://sites.mitre.org/armychallenge/to enter. All submissions must conform to the requirements set forth on https://sites.mitre.org/armychallenge/.