Sim-to-real gap¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The fidelity contract. Every synthetic OOD (out-of-distribution) score EMMA (Electromagnetic Multi-task Model Assessment) reports is paired with the same model’s score on a real-capture OOD subset (EMMA-REAL-OOD). The difference, in percentage points, is the sim-to-real gap, computed by SimToRealGap. A large gap means the model fit rfgen’s physics model rather than learned signal competence, and the gap is reported alongside the synthetic score rather than folded into it. This converts the field’s oldest critique of synthetic RF (radio-frequency) data into a first-class metric. The narrative motivation lives in Sim-to-real gap.

The gap¶

Each task has a normalized score on the synthetic split (\(\hat{m}_{\text{synth}}\)) and on the real-capture OOD subset (\(\hat{m}_{\text{real}}\)), both mapped to \([0, 1]\) by the same normalization as OOD-avg (see Aggregation). The sim-to-real gap is the difference, scaled to percentage points:

\[ \Delta_{\text{s2r}} = 100 \cdot \bigl(\hat{m}_{\text{synth}} - \hat{m}_{\text{real}}\bigr) \]

Units. Percentage points (pp).
Sign convention. A positive gap means the synthetic score exceeds the real-capture score; the model performed better on rfgen than on real captures.
Interpretation. A gap near zero means the model’s competence transfers from synthetic scenes to real captures. A large positive gap means the model exploited something specific to rfgen that does not hold in the real world.

What the gap is and is not¶

The real-capture score does not replace the synthetic score in OOD-avg. The two answer different questions: the synthetic score measures controlled OOD transfer; the gap measures fidelity to reality. The gap is a column on the leaderboard, reported alongside OOD-avg, never averaged into it. A model can rank first on OOD-avg and carry a large gap; both numbers are read together.

The real-capture OOD subset is small by design (on the order of thousands of scenes, drawn from the three public testbeds: Colosseum, POWDER, COSMOS). It exists to measure fidelity, not to train on. Models do not train on the real subset; they are only evaluated on it. The subset is frozen at release so the gap is comparable across submissions.

Threshold principle¶

A single threshold that flags an actionable gap (for example, “a gap above \(T\) percentage points is a fidelity failure”) is proposed-contract. The principle for the threshold: it should be set at the point where the gap is large enough that the synthetic rank is no longer a trustworthy proxy for real-world competence, calibrated against the first real-capture scores once the RealCaptureAdapter set lands. The threshold is frozen at release and reported with the column; it is never tuned per submission. Until the real-capture scores exist, no threshold is asserted.

Real-capture data flow¶

The real captures are ingested through RealCaptureAdapter implementations and scored with the same frozen backbone and the same per-task metrics as the synthetic split. See Architecture.

synthetic scenes (rfgen) ──► FrozenBackbone ──► score_synth   ┐
                                                              ├─► SimToRealGap
real captures (testbeds) ──► same backbone  ──► score_real   ┘
                                                              │
                                                              ▼
                                          leaderboard column (percentage points)

References¶

“RF-Analyzer,” 2026, arXiv:2605.04676. Concedes that sim-to-real generalization breaks in low-SNR (signal-to-noise ratio) and OOD regimes; the load-bearing sim-to-real caveat this metric surfaces. (verify)
Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Paired in-distribution and OOD evaluation as a benchmark contract.
Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” NeurIPS 2019, arXiv:1906.02530. Predictive performance under shift. (verify)