Signal as a modality

Pre-launch / design spec

The task set and aggregation below are proposals for v0.1. Specifics freeze with the first data release.

Overview

Each foundation-model modality has a shared, governed benchmark that lets one pretrained backbone be read out across many tasks and compared on a common scale: GLUE and SuperGLUE for language, ImageNet for vision, SUPERB (Speech processing Universal PERformance Benchmark) for speech, and MLPerf for systems. EMMA (Electromagnetic Multi-task Model Assessment) is that benchmark for a new modality, electromagnetic (radio-frequency) signal. One frozen backbone, lightweight task-specific readouts, one aggregated score.

EMMA’s distinct contribution is forced by the signal domain. The headline metric is generalization, not in-distribution accuracy. The canonical input is the raw phase-coherent multi-antenna I/Q (in-phase and quadrature) that direction-finding physically requires. And, because large real signal captures are scarce, every synthetic score is paired with a sim-to-real gap.

Place In The System

This page is the framing the rest of the Concepts section assumes. The concrete task list lives in Tasks; the aggregation math lives in Generalization; the pipeline that wires it together lives in Architecture (developer guide).

The shared template, and where EMMA diverges

The benchmarks above share a template: a frozen upstream backbone, a lightweight readout per task, and a single aggregated score. EMMA keeps all three.

Shared template

EMMA’s divergence

Backbone

One frozen upstream model, read out, not fine-tuned

Kept

Readouts

One lightweight head per task

Kept

Score

One normalized aggregate

Kept

Headline metric

In-distribution accuracy

Transfer under controlled out-of-distribution axes

Input

The modality’s raw native data

Raw phase-coherent multi-antenna I/Q

Honesty check

Real captures are abundant

A sim-to-real gap column, because signal captures are scarce

Operator

Usually academic

Neutral governance, because the operator also builds signal models

What EMMA keeps

The shared template rests on three commitments EMMA adopts directly.

  1. One frozen backbone. Weights are frozen at evaluation time. Only the readout head trains. This isolates representation quality from head capacity. See FrozenBackbone.

  2. Lightweight readouts. Each task attaches a small head (for example AngularRegressionHead, ClassificationHead). If a head must be large to score well, the backbone is not doing the work.

  3. One aggregated score. Per-task scores are normalized and averaged into a single headline number by OODAvg, so a model cannot win by specializing in one column.

Why a new modality benchmark

Signal is a first-class AI modality alongside language, vision, and speech: it carries information about emitters, motion, spectrum behavior, and physical context that no camera or microphone sees. Yet no shared surface lets one signal backbone be read out across tasks and compared across groups. Every existing asset is single-task (RadioML, MSTAR (Moving and Stationary Target Acquisition and Recognition)), single-antenna, or a derived representation (DeepMIMO at the CSI (channel state information) level). EMMA fills that gap. See Data model.

Design notes

  • Do not lead with saturated tasks. GLUE saturated and forced the move to SuperGLUE; RadioML is saturated. EMMA excludes AMC (automatic modulation classification) from the aggregate and keeps it only as a continuity column. See Tasks.

  • Generalization is the score. Following WILDS, distribution shift is the headline, not an appendix. Leave-one-environment-out is a first-class protocol. See Generalization.

  • A tribe contributes a task. Following BIG-bench, the task registry is designed to grow by community contribution once the v0.1 inner ring is proven. See Tasks.

  • The score is testable. Under leave-one-environment-out, a model that overfits the channel statistics of one environment scores no better than the per-task chance baseline on the held-out environment, averaged over held-out environments. See Generalization.

References

  • Wang et al., “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding,” EMNLP 2018, arXiv:1804.07461. A per-modality, one-backbone-many-tasks benchmark; saturation signaled the move to SuperGLUE.

  • Wang et al., “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems,” NeurIPS 2019, arXiv:1905.00537.

  • Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” IJCV 2015, arXiv:1409.0575. The vision-modality benchmark that became a standard.

  • Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. The closest structural analog: frozen backbone, lightweight readouts, one aggregate.

  • Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Consortium-governed, neutral-operator benchmarking.

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Generalization-first precedent.

  • Srivastava et al., “Beyond the Imitation Game (BIG-bench),” 2023, arXiv:2206.04615. The tribe-contributes-a-task adoption mechanism.

See Also

  • Tasks: the five task pillars and why localization is the flagship.

  • Generalization: the OOD (out-of-distribution) axes and the transfer score.

  • Data model: why raw multi-antenna I/Q is non-negotiable.

  • Sim-to-real gap: the honesty check signal benchmarks need.

  • Neutrality: the ring-fence and third-party re-scoring.