Generalization

Pre-launch / design spec

The protocol below is the design target for v0.1. Specifics (axis choices, normalization constants) will be finalized and frozen with the first data release.

The methodology of EMMA (Electromagnetic Multi-task Model Assessment) rests on one principle: the score is transfer, not in-distribution accuracy. Everything below follows from making that rigorous and reproducible.

The OOD axes

Generalization is the score, so the out-of-distribution (OOD) axis along which a model is stressed is the contract. Each task holds out one controlled dimension at evaluation time:

  • Environment. A channel environment unseen at train time (urban multipath held out against rural near-line-of-sight, and vice versa). The default protocol axis for localization.

  • Device. Every capture of one device held out, isolating device identity from capture session and receiver front-end. The E-ID-FP axis.

  • Frequency band. A band unseen at train time (for example 2.4 GHz held out against 5.8 GHz). Layered onto LoS / NLoS (line-of-sight / non-line-of-sight) classification.

  • SNR (signal-to-noise ratio) regime. A noise-floor regime unseen at train time. The E-ID-AMC axis, and a stress axis across tasks.

Why transfer rather than accuracy: in-distribution accuracy is what every saturated single-task dataset already measures, and a model can score high on it while failing the moment the channel, device, or band changes. Scoring only transfer forces the benchmark to measure the property that survives deployment. See OOD protocol for the exact fold-construction contract.

Leave-one-environment-out vs leave-one-unit-out

For each task, the training split spans a set of environments \(\mathcal{E}_{\text{train}}\). Evaluation holds out one environment \(e^\* \notin \mathcal{E}_{\text{train}}\) at a time and reports the metric on \(e^\*\). The reported score is the mean across held- out environments:

\[ \text{score} = \frac{1}{|\mathcal{E}_{\text{eval}}|} \sum_{e \in \mathcal{E}_{\text{eval}}} m(\text{model}, e) \]

v0.1 ships with at least two environments (E-URBAN, E-RURAL), so a model that overfits the channel statistics of one cannot hide. Per-task axes layer on top (unseen device, frequency band, SNR regime) - see Tasks.

Leave-one-environment-out answers “does the model transfer across channel conditions?” Leave-one-unit-out answers a different question: “does the model identify a device it has never seen, without confounding the capture session?” The two are not interchangeable; each task pins the axis that matches what it measures.

Per-task metrics

Task

Primary metric

Direction

E-LOC-AOA

Mean angular error (MAE, degrees)

lower

E-LOC-LOS

Accuracy (balanced)

higher

E-ID-DRONE

AUC (area under the receiver operating characteristic curve)

higher

E-ID-FP

Top-1 accuracy (leave-one-unit-out)

higher

E-ID-AMC

Accuracy (continuity, not aggregated)

higher

E-LOC-AOA is scored as continuous regression, not binned classification - binning lets a model hide a systematic angular bias behind class accuracy.

Aggregation

The headline OOD-avg normalizes each scored task to \([0,1]\) and averages. Lower-is-better metrics are inverted first. With min/max anchors \((m_{\min}, m_{\max})\) frozen per task at release:

\[\begin{split} \hat{m} = \begin{cases} \dfrac{m - m_{\min}}{m_{\max} - m_{\min}} & \text{higher is better} \\[6pt] \dfrac{m_{\max} - m}{m_{\max} - m_{\min}} & \text{lower is better} \end{cases} \qquad \text{OOD-avg} = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \hat{m}_t \end{split}\]

where \(\mathcal{T}\) is the set of aggregated v0.1 tasks (E-LOC-AOA, E-LOC-LOS, E-ID-DRONE, E-ID-FP). Automatic modulation classification (E-ID-AMC) is reported but excluded - it is saturated and single-antenna-solvable, so including it would dilute the signal that matters. See Aggregation for the exact normalization anchors and the aggregator contract.

Sim-to-real gap

Every synthetic score is paired with the same model’s score on the real-capture OOD subset (EMMA-REAL-OOD). The gap is reported in percentage points:

\[ \Delta_{\text{s2r}} = 100 \cdot \big(\hat{m}_{\text{synth}} - \hat{m}_{\text{real}}\big) \]

A large \(\Delta_{\text{s2r}}\) means the model has fit rfgen’s physics model rather than learned signal competence. This converts the field’s oldest critique of synthetic radio-frequency (RF) data into a first-class metric. See Sim-to-real gap for the full treatment.

Reproducibility

Scenes are content-hashed from a frozen rfgen commit, config, and seed (see Datasets). The held-out split is a frozen secret seed; the public train/dev splits are re-derivable by anyone. Every leaderboard row carries the model’s prediction bundle hash and the data release hash, so a result is an auditable (model, data, metric) triple, not a self-reported number. See Reproducibility for the content-hash construction.

References

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. The leave-one-X-out framing and generalization-first precedent.

  • Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” NeurIPS 2019, arXiv:1906.02530. Predictive performance under controlled shift.

  • Brodersen et al., “The balanced accuracy and its posterior distribution,” ICPR 2010. Balanced accuracy for E-LOC-LOS. (verify)

  • Davis and Goadrich, “The relationship between Precision-Recall and ROC curves,” ICML 2006. AUC grounding for E-ID-DRONE. (verify)

See Also