Sim-to-real gap

Pre-launch / design spec

The real-capture OOD subset and the gap metric are proposed for v0.1. The pinned testbed captures and normalization anchors finalize with the first data release.

Overview

Every synthetic score EMMA (Electromagnetic Multi-task Model Assessment) reports is paired with the same model’s score on a real-capture OOD (out-of-distribution) subset. The difference, in percentage points, is the sim-to-real gap. A large gap means the model fit rfgen’s physics model rather than learned signal competence, and it is reported alongside the synthetic score rather than buried in an appendix. This converts the field’s oldest critique of synthetic radio-frequency (RF) data into a first-class metric.

Place In The System

The sim-to-real gap is computed by SimToRealGap after OODAvg normalizes per-task scores. It is a column on the leaderboard, not part of the aggregate score. The real-capture data it depends on is ingested through RealCaptureAdapter implementations (Colosseum, POWDER, COSMOS). See Architecture.

Boundaries

  • Owned by EMMA: the gap metric (SimToRealGap), the real-capture adapters, and the reporting column.

  • Owned by external testbeds: the real captures themselves (Colosseum at Northeastern, POWDER and COSMOS at NSF PAWR).

  • Not in scope: changing the synthetic score. The gap is reported alongside it; the headline OOD-avg is computed on synthetic data only.

Data Flow

synthetic scenes (rfgen) ─► FrozenBackbone ─► score_synth   ┐
                                                            ├─► SimToRealGap
real captures (testbeds) ─► same backbone  ─► score_real   ┘
                                                            │
                                                            ▼
                                          leaderboard column (percentage points)

The Gap

Each task has a normalized score on the synthetic split (\(\hat{m}_{\text{synth}}\)) and on the real-capture OOD subset (\(\hat{m}_{\text{real}}\)), both mapped to a 0 to 1 range by the same normalization. The sim-to-real gap is the difference, scaled to percentage points:

\[ \Delta_{\text{s2r}} = 100 \cdot \big(\hat{m}_{\text{synth}} - \hat{m}_{\text{real}}\big) \]

A gap near zero means the model’s competence transfers from synthetic scenes to real captures. A large positive gap means the model exploited something specific to rfgen that does not hold in the real world. The aggregate gap is computed by SimToRealGap; see Aggregation for the per-task normalization.

Why This Metric Exists

Synthetic RF data has been critiqued for over a decade: models trained on it fail to generalize to real captures, particularly in low-SNR (signal-to-noise ratio) and OOD regimes. The standard response is to argue the gap is small. EMMA instead measures it on every submission and prints it next to the score. RF-Analyzer (2026) concedes that sim-to-real generalization breaks precisely in the regimes EMMA scores; that concession is the load-bearing motivation for making the gap a column rather than a footnote.

The real-capture OOD subset (EMMA-REAL-OOD) is small by design (on the order of thousands of scenes, drawn from the three public testbeds). It exists to measure fidelity, not to train on. Models do not train on the real subset; they are only evaluated on it.

Minimal Example

from emma.harness import Evaluator
from emma.metrics import SimToRealGap

evaluator = Evaluator.for_model("my-fm@v1")
synth = evaluator.score_subset("EMMA-LOC-v0.1", split="dev")    # synthetic
real  = evaluator.score_subset("EMMA-REAL-OOD-v0.1")            # real captures
gap = SimToRealGap().compute(synth.ood_avg, real.ood_avg)       # percentage points

Design Notes

  • Gap, not replacement. The real-capture score does not replace the synthetic score in OOD-avg. The two answer different questions: the synthetic score measures controlled OOD transfer; the gap measures fidelity to reality.

  • Small and frozen. The real subset is small enough to score cheaply and frozen at release so the gap is comparable across submissions.

  • The critique becomes the metric. A reviewer who distrusts synthetic RF can read the gap column directly. A model that cannot cross from rfgen to Colosseum is visible regardless of its synthetic rank.

References

  • “RF-Analyzer,” 2026, arXiv:2605.04676. Concedes that sim-to-real generalization breaks in low-SNR and OOD regimes; the load-bearing sim-to-real caveat this metric surfaces.

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Paired in-distribution and OOD evaluation as a benchmark contract.

  • Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” NeurIPS 2019, arXiv:1906.02530. Predictive performance under shift.

See Also