Leaderboard

Pre-launch / design spec

No leaderboard is live yet. The schema and sample rows below show the intended shape. Rows marked illustrative are placeholder numbers, not real results.

The EMMA (Electromagnetic Multi-task Model Assessment) leaderboard scores models on the v0.1 task set, each under its out-of-distribution (OOD) axis, and reports a sim-to-real gap column. The private holdout set is never released - scores are produced by prediction submission (you submit predictions; EMMA scores them). At v1 this moves to sandboxed code submission.

v0.1 schema

Rank

Model

Submitter

E-LOC-AOA (MAE °↓)

E-LOC-LOS (Acc ↑)

E-ID-DRONE (AUC ↑)

E-ID-FP (Top-1 ↑)

E-ID-AMC† (Acc ↑)

OOD-avg ↑

sim→real gap

-

Superpose seed (baseline)

Superpose

6.4

0.91

0.93

0.78

0.62

0.79

4.1 pp

-

illustrative

-

5.1

0.94

0.95

0.82

0.60

0.83

3.0 pp

-

illustrative

-

7.9

0.88

0.90

0.71

0.66

0.76

6.5 pp

Metric acronyms: MAE is mean angular error, AUC is area under the ROC (receiver operating characteristic) curve, AMC is automatic modulation classification, Acc is accuracy, pp is percentage points.

† E-ID-AMC (modulation classification) is a continuity column, reported but excluded from the aggregate. OOD-avg is the normalized mean of the four v0.1 scored tasks; lower-is-better metrics (angular error) are inverted before normalization - see Aggregation and the aggregation reference.

Lower angular error and a smaller sim→real gap are better. A model that tops the synthetic columns but carries a large sim-to-real gap is not winning - it has memorized rfgen, not learned signal physics.

Each row is a LeaderboardRow carrying the data-release hash and the per-task scores, with the per-task prediction-bundle hashes on its linked RunManifest, so a result is an auditable (model, data, metric) triple. The exact schema is documented at Leaderboard row.

Submission

At v0.1, submission is results-only via prediction file:

# produce predictions on the (public) dev set, then submit for holdout scoring
emma eval --task E-LOC-AOA --model ./my-backbone --out preds/E-LOC-AOA.parquet
emma submit --leaderboard v0.1 --predictions preds/ --model "my-fm@v1"

EMMA scores your predictions against the private holdout and posts the row. The model itself never leaves your machine at v0.1; sandboxed code submission arrives at v1.

Note

Superpose’s own model participates under the same rules and is re-scored by a third party on every release. A deliberately-beatable open baseline (Apache-2.0) is seeded on the board from day one - see Neutrality.

References

  • Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. The single aggregated score the OOD-avg column generalizes.

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Results-only evaluation under controlled distribution shift.

  • Hanna and Hussain, “Robust Low-SNR Modulation Classification,” 2026, arXiv:2605.27673. Grounds that AMC is saturated, hence a continuity column excluded from the aggregate.

See Also