Datasets validation

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

Scope: the six-lens validation of the 15 dataset recipes catalogued in Datasets, with depth concentrated on the five v0.1 recipes (EMMA-LOC-v0.1, EMMA-DRONE-v0.1, EMMA-FP-v0.1, EMMA-AMC-v0.1, EMMA-REAL-OOD-v0.1). Later-version recipes (EMMA-RWAVE-v0.2, EMMA-UAVDOP-v0.2, EMMA-BEAM-v0.2, EMMA-POS-v1, EMMA-CSI-v1, EMMA-SC-*, EMMA-S2T-*) carry construct and literature coverage plus their open dependency flags; their empirical and methodology sections land when their recipes pin.

1. Purpose and construct

A dataset recipe is a pinned rfgen configuration plus a label-extraction map; the generator, not a static file, is the test set. Construct validity asks, per recipe, whether the pin actually produces the scene distribution its task claims, and whether LabelExtractor reads the right MetadataPath to the right label. The central construct rule is the boundary: a recipe pins rfgen output via rfgen.{commit, config, array_geometry, num_rx} (the config references a composed rfgen GenerationConfig that already pins the emitter_zoo, channel, scene, placement, label, storage, executor, and run groups), partitions integer seeds via splits.<Split> as a SeedRange (start, stop, count), and declares each task label as a MetadataPath into rfgen metadata under label_extraction. It never re-specifies emitter, channel, or propagation physics; if a recipe appears to define an emitter or a channel model, it has crossed into rfgen and the construct is broken.

Recipe-to-rfgen-schema consistency

Each recipe pins a composed rfgen GenerationConfig by path under rfgen.config, pins receiver geometry directly under the rfgen block via array_geometry (an ArrayGeometry member: ULA (uniform linear array) or URA (uniform rectangular array)) and num_rx, and reaches rfgen emitter families (comms, radar, drone_rf) through the composed config. The construct check is that every recipe field maps to a documented rfgen config surface or enum member, not to a re-specification of physics.

  • EMMA-LOC-v0.1 pins rfgen.config (a composed rfgen GenerationConfig over comms emitters, Sionna UMa and RMa channels, and the rfgen label group), rfgen.array_geometry (URA), and rfgen.num_rx (8); its label_extraction reads emitters[].aoa_deg for the AoA (angle-of-arrival) regression target and emitters[].los_flag for the LoS / NLoS (line-of-sight / non-line-of-sight) binary label. Construct-valid: it labels rfgen output via MetadataPath, it does not define the ray tracer.

  • EMMA-DRONE-v0.1, EMMA-FP-v0.1, EMMA-AMC-v0.1 follow the same pattern across their respective families and label paths.

  • EMMA-REAL-OOD-v0.1 is not an rfgen recipe; its construct is the RealCaptureAdapter ingest contract, with no physics pin at all.

Sharpest construct threats

(Severity: high, architectural) The recipes for EMMA-RWAVE-v0.2, EMMA-UAVDOP-v0.2, and the EMMA-S2T-* v2 pair depend on rfgen features not yet shipped (radar-emitter coverage, real-time micro-Doppler, the Phase 2 annotator). They are flagged as placeholders on their recipe pages and cannot carry a construct claim until those features land. Treating them as complete would silently assert physics EMMA cannot yet pin.

(Severity: high, architectural) EMMA-FP-v0.1 construct validity is conditional on a rigorous leave-one-unit-out recipe that isolates device identity from capture session and receiver front-end. If the rfgen FingerprintConfig does not randomize the receiver chain across units, the score measures channel or session features rather than device identity. See Tasks validation.

2. Mathematical fidelity

Recipes do not define metric formulas; that audit lives in Metrics validation. The dataset-side math is the seed and split accounting.

  • Seed determinism. A scene is fully determined by (rfgen commit, resolved config, seed). The recipe’s splits field maps each Split member (TRAIN, DEV, HOLDOUT) to a SeedRange (start, stop, count); the three closed integer intervals [start, stop] must be pairwise disjoint, and the realized scene count per split must equal count (enforced by the loader). For a contiguous range, count equals stop - start + 1. (Severity: medium) Open: a contract test that re-derives each split’s SeedRange and asserts zero pairwise overlap, plus agreement between count and the interval length when the range is contiguous.

  • Content-hash composition. The SceneManifest content_hash and the DataReleaseManifest content_hash must compose deterministically (release hash over the manifest plus its referenced scene hashes). (Severity: low) Covered by Reproducibility and the content-hash algorithm reference.

  • Label-path resolution. The label_extraction map names rfgen metadata paths (for example emitters[].aoa_deg) via the MetadataPath grammar <container>[].<rfgen_key>. A path that does not resolve on a single realized scene yields an empty label for that scene; when every scene in a split misses the path, the loader raises RecipeMismatchError rather than scoring silently. The EmitterRecord promotion of aoa_deg and los_flag from the raw rfgen metadata bag is the path-target contract. (Severity: medium) Open: a loader test that every label_extraction path resolves on a sample scene and that a split-wide miss raises RecipeMismatchError (see section 5).

3. Empirical realism

Each v0.1 recipe depends on a pinned rfgen configuration; rfgen’s own validation reports are the primary evidence for physical credibility. Where rfgen has published validation against real captures or classical estimators, this section cites it; where it has not, the gap is named.

  • Inter-antenna phase coherence (load-bearing). EMMA-LOC-v0.1 (AoA), EMMA-BEAM-v0.2, and EMMA-POS-v1 all read the phase difference between antennas. If the synthetic array’s per-element phase is not physically consistent, a model can score well on a phase artifact rather than a true direction. (Severity: high, open) rfgen’s inter-antenna phase coherence is not yet independently verified in a published report. The proposed verification is a side-by-side against a classical MUSIC / ESPRIT estimator on identical arrays and snapshots (section 5). This is the single largest empirical-realism threat to the flagship and is cross-linked from Tasks validation.

  • Urban and rural channel realism. E_URBAN (Sionna UMa) and E_RURAL (Sionna RMa) inherit their physical credibility from Sionna’s own validation against ray-tracing and measurement campaigns. (Severity: medium) Open: cite rfgen’s channel-validation report once published, rather than asserting realism by recipe design.

  • Real-capture fidelity. EMMA-REAL-OOD-v0.1 is the empirical anchor for sim-to-real realism, but it is wired at v0.1 as fidelity only and its coverage is uneven (drone-RF and device-fingerprint real captures are scarce). The RF-Analyzer result that sim-to-real generalization breaks in low-SNR (low signal-to-noise ratio) and OOD (out-of-distribution) regimes is the load-bearing caveat. (Severity: medium)

4. Literature grounding

The canonical source per recipe family, drawn from the master reading list. No new literature entries are required for this report; every citation below already appears there.

  • Localization (EMMA-LOC-v0.1, EMMA-POS-v1). Schmidt (MUSIC) and Roy and Kailath (ESPRIT) ground that AoA is a continuous angular quantity and supply the gold-standard estimators for the phase-coherence probe. Van Trees is the textbook of record.

  • Drone-RF (EMMA-DRONE-v0.1). Media Inhof et al. (DroneRF) and Coluccia et al. (DroneDetect / RFUAV) are the single-antenna benchmarks EMMA moves beyond.

  • Fingerprinting (EMMA-FP-v0.1). The DARPA RFMLS program, Bihl, Bauer, and Temple, and “No Radio Left Behind” ground the leave-one-unit-out construct.

  • AMC (EMMA-AMC-v0.1). O’Shea et al. (RadioML lineage), the DeepSig errata note, and Hanna and Hussain (saturation) ground the continuity-column exclusion.

  • Beam and CSI (channel state information; EMMA-BEAM-v0.2, EMMA-CSI-v1). 3GPP TR 38.843 names the use cases without a dataset; Alkhateeb (DeepMIMO) and Djordjevic, Ali, and Alkhateeb (LWM) are the CSI camp the bridge targets.

  • Radar (EMMA-RWAVE-v0.2, EMMA-UAVDOP-v0.2). Skolnik (textbook) and Chen et al. (micro-Doppler).

  • Scene understanding (EMMA-SC-*). Le Roux et al. (SI-SDR), Davis and Goadrich (AUC / AUROC), Ovadia et al. (uncertainty under shift).

  • Signal-to-text (EMMA-S2T-*). Papineni (BLEU, Bilingual Evaluation Understudy), Banerjee and Lavie (METEOR, Metric for Evaluation of Translation with Explicit ORdering), Vedantam (CIDEr, Consensus-based Image Description Evaluation), Rajpurkar (SQuAD).

  • Real-capture and sim-to-real (EMMA-REAL-OOD-v0.1). The Colosseum, POWDER, and COSMOS testbeds; “RF-Analyzer” for the sim-to-real break.

  • Generator and libraries. NVIDIA Sionna (the ray tracer rfgen builds on) and TorchSig (the modulation library rfgen composes).

(Severity: low) Several citations carry (verify) in the reading list; none are presented as fact here until verified.

5. Experimental methodology and planned tests

Tests are written so an implementer can port them directly to tests/validation/datasets/ once src/emma/ lands. Each entry records the falsifiable claim, the failure mode, the design, the sample size, the statistical test, the tolerance, and the gold-standard reference. Until src/emma/ and the pinned rfgen recipes are runnable, each proposed gate ships with a pytest.skip marker and a one-line rationale pointing to the missing dependency: the loader contract for the label-path and split tests, and the pinned rfgen recipe for the phase-coherence and sim-to-real tests. The pre-implementation warning at the top of this page is the standing rationale.

Phase-coherence gold standard - tests/validation/datasets/test_phase_coherence.py

  • Falsifiable claim. A classical estimator recovers known emitter directions from rfgen scenes on the pinned array, within a bounded angular error.

  • Failure mode. The estimator fails to recover directions on rfgen scenes but succeeds on a synthetic analytic array; this flags a generation bug, not a model bug.

  • Design. Run MUSIC and ESPRIT on EMMA-LOC-v0.1 scenes using the same 8-element URA and snapshots; compare estimated AoA to the rfgen ray tracer’s ground-truth aoa_deg metadata.

  • Sample size. Minimum 2000 scenes across both environments.

  • Statistical test. Wrapped mean angular error between the classical estimate and the ground-truth angle, with a paired bootstrap 95% confidence interval.

  • Tolerance. Classical-estimator mean angular error below a fixed delta (pinned with the first data release). A failure here blocks scoring of E-LOC-AOA, E-CH-BEAM, and E-LOC-POS.

  • Gold-standard reference. MUSIC (Schmidt) and ESPRIT (Roy and Kailath) via scipy and Sionna primitives.

Split non-leakage - tests/validation/datasets/test_split_nonleakage.py

  • Falsifiable claim. The TRAIN, DEV, and HOLDOUT seed ranges are disjoint, the realized scene count per split equals SeedRange.count, and the holdout seeds are absent from every public release artifact.

  • Failure mode. Two SeedRange intervals overlap, a realized scene count disagrees with count, or a holdout seed appears in a dev manifest.

  • Design. Parse each DataReleaseManifest, collect the per-split SeedRange (start, stop, count) from the recipe, assert pairwise-empty intersection of the closed [start, stop] intervals, assert the realized scene count per split equals count, and scan public manifests for holdout seeds.

  • Statistical test. Set-membership assertion (deterministic).

  • Tolerance. Zero pairwise seed overlap; realized scene count equals SeedRange.count per split; zero holdout seeds in public artifacts.

  • Gold-standard reference. The recipe’s splits map and the release manifest’s split_assignments.

Label-path resolution - tests/validation/datasets/test_label_path_resolution.py

  • Falsifiable claim. Every MetadataPath in a v0.1 synthetic recipe’s label_extraction map resolves on a realized scene, and a split-wide miss raises RecipeMismatchError.

  • Failure mode. rfgen renames or drops a metadata field a recipe targets (canonical case: emitters[].aoa_deg), so per-scene labels silently empty across an entire split.

  • Design. For each v0.1 synthetic recipe, regenerate a sample of scenes at the pinned rfgen.commit and rfgen.config; resolve each label_extraction path on each scene via LabelExtractor; assert a non-empty label tensor per scene, and assert that a synthetic all-miss split raises RecipeMismatchError. Real-capture-only recipes (for example EMMA-REAL-OOD-v0.1) carry no label_extraction map and are out of scope.

  • Sample size. 100 scenes per split per synthetic recipe.

  • Statistical test. Set-membership and dtype assertion (deterministic).

  • Tolerance. Zero unresolved paths; zero silently empty splits.

  • Gold-standard reference. The rfgen metadata schema pinned by the recipe and the EmitterRecord fields (aoa_deg, los_flag) promoted onto the SceneManifest.

Leave-one-environment-out power - tests/validation/datasets/test_loeo_power.py

  • Falsifiable claim. With two synthetic environments, per-environment scores are reported alongside every aggregate so a single outlier environment cannot dominate.

  • Failure mode. Only the aggregate is reported; a model that overfits one environment scores near-chance on the other but the aggregate hides it.

  • Design. Assert every scored recipe emits per-environment (E_URBAN, E_RURAL) breakdowns in addition to the aggregate; run a jackknife (drop each environment in turn) to bound the aggregate’s sensitivity.

  • Statistical test. Jackknife range of the aggregate across environment drops.

  • Tolerance. The v0.1 aggregate is labeled a comparative ranking, not a population estimate; the jackknife range is reported as the uncertainty.

  • Gold-standard reference. The two-environment structure in Environment reference.

Sim-to-real gap threshold - tests/validation/datasets/test_sim_to_real.py

  • Falsifiable claim. The sim-to-real gap between a synthetic-trained model and its EMMA-REAL-OOD-v0.1 score is reported and bounded.

  • Failure mode. The gap is silently large in low-SNR or OOD regimes, masking that synthetic gains do not transfer.

  • Design. For each synthetic recipe with a real-capture counterpart, compute the score gap on matched tasks; stratify by SNR regime.

  • Statistical test. Paired bootstrap CI on the gap.

  • Tolerance. Gap threshold pinned with the first real-capture release; the expected failure region (low SNR, OOD) is reported explicitly per the RF-Analyzer caveat.

  • Gold-standard reference. RealCaptureAdapter captures; the RF-Analyzer low-SNR break.

Open methodology question

(Severity: high, architectural, deferred to open questions) Leave-one-environment-out with only two environments gives \(N = 2\) for the environment-level statistic, which has essentially no power to distinguish models at the environment stratum. The mitigations are the same as in Tasks validation: add a third environment when the recipe allows, report per-environment scores, and treat the v0.1 aggregate as comparative ranking under a fixed protocol.

6. Robustness boundaries

Each boundary is a claim about where validity breaks, paired with a proposed probe.

  • Recipe drift on rfgen breaking changes. (Severity: high) A recipe pins a single rfgen commit. If rfgen ships a breaking change to an emitter, channel, or metadata schema, an older recipe may fail to re-derive or may silently produce different scenes. Probe: a re-derivation test that regenerates a sample of each recipe at its pinned commit and compares the content_hash to the release manifest; a mismatch flags drift. The mitigation is that a new release ships a new commit pin and hash without invalidating the old.

  • Inter-antenna phase fidelity. (Severity: high) Covered in sections 3 and 5; the MUSIC / ESPRIT gold-standard test doubles as the phase-coherence probe.

  • Real-capture coverage gaps. (Severity: medium) Drone-RF and device-fingerprint real captures are scarce in the public testbed set, so the sim-to-real gap for EMMA-DRONE-v0.1 and EMMA-FP-v0.1 is asserted by recipe design, not measured, at v0.1. Probe: a coverage matrix per task once the adapters land; tasks below a capture-count floor report no sim-to-real gap rather than an underpowered one.

  • Real-capture adapter mismatch. (Severity: medium) A RealCaptureAdapter could populate the SceneManifest or the multi-antenna I/Q layout (num_rx, 2, N) differently from rfgen, breaking the sim-to-real comparison; real captures also lack ground truth for emitters[].aoa_deg and emitters[].los_flag. Probe: a contract test that each adapter’s list_scenes and load_iq outputs match the SceneManifest schema and the EMMADataset I/Q invariant, with the sim-to-real gap column reading None for labels real captures cannot provide.

  • Unshipped-feature dependencies. (Severity: medium) EMMA-RWAVE-v0.2, EMMA-UAVDOP-v0.2, and EMMA-S2T-* depend on rfgen features not yet shipped. Probe: a recipe-completeness gate that refuses to score a task whose recipe carries an open dependency flag.

  • Label-path drift. (Severity: medium) If rfgen renames a metadata field a recipe’s label_extraction map targets (canonical case: emitters[].aoa_deg), per-scene labels empty silently and a split-wide miss raises RecipeMismatchError. Probe: the label-path resolution test in section 5.

  • Small-N environment statistics. (Severity: high) Covered in section 5; the two-environment aggregate is fragile and must ship with per-environment scores.

7. Synthesis

Load-bearing claims supported. Every recipe pins rfgen config groups by name rather than re-specifying physics, so the recipe-to-rfgen boundary holds. The v0.1 recipes are construct-valid conditional on the open risks below. The continuity-column exclusion of AMC and the leave-one-unit-out construct for fingerprinting are literature-grounded. The content-hash and seed-determinism model is sound in principle.

Gaps surfaced. (1) rfgen inter-antenna phase coherence is unverified and is the single largest threat to the flagship localization and beam recipes. (2) Leave-one-environment-out at \(N = 2\) environments is underpowered; the v0.1 aggregate is a comparative ranking, not a population estimate. (3) Three later-version recipes depend on rfgen features not yet shipped and are placeholders. (4) Real-capture coverage is uneven, so sim-to-real realism for drone-RF and fingerprinting is asserted, not measured. (5) The EMMA-FP-v0.1 receiver-chain randomization is not pinned, leaving the session-confound door open.

Recommended actions. (1) Land the MUSIC / ESPRIT gold-standard side-by-side as the first tests/validation/datasets/ test; it validates the generator, the localization recipe, and the phase-coherence assumption together. (2) Pin the EMMA-FP-v0.1 receiver-chain and session randomization in the recipe before fingerprinting is scored. (3) Add a recipe-completeness gate that blocks scoring for recipes with open dependency flags. (4) Wire the real-capture adapters with a contract test on SceneManifest and I/Q-layout fidelity, plus a per-task coverage matrix, before claiming empirical realism beyond positioning and AMC. (5) Land the split non-leakage and label-path resolution tests in section 5 as part of the loader contract; both ship skip-with-rationale until src/emma/ and the pinned rfgen recipes are runnable.

Top three findings.

  1. (high) rfgen phase-coherence fidelity is unverified and load-bearing for EMMA-LOC-v0.1, EMMA-BEAM-v0.2, and EMMA-POS-v1; the MUSIC / ESPRIT gold-standard test is the proposed first verification.

  2. (high) EMMA-FP-v0.1 construct validity is conditional on receiver-chain and session randomization that the recipe does not yet pin; scoring before that pin measures session features, not device identity.

  3. (high) Three later-version recipes (EMMA-RWAVE-v0.2, EMMA-UAVDOP-v0.2, EMMA-S2T-*) depend on rfgen features not yet shipped and must be gated as incomplete until those features land.

References

  • Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE Trans. Acoustics, Speech, Signal Processing 1986, DOI:10.1109/TASSP.1986.1164830. MUSIC; the gold-standard estimator for the phase-coherence probe. (verify)

  • Roy and Kailath, “ESPRIT: Estimation of signal parameters via rotational invariance techniques,” IEEE Trans. Acoustics, Speech, Signal Processing 1989, DOI:10.1109/29.32276. Second classical estimator for the phase-coherence probe. (verify)

  • NVIDIA Sionna: Hoydis et al., “Sionna: An Open-Source Library for Next-Generation Physical Layer Research,” 2023, arXiv:2203.11854. The UMa and RMa channel models and the estimator primitives. (verify)

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. The leave-one-X-out framing and small-N caveat. (verify)

  • “RF-Analyzer,” 2026, arXiv:2605.04676. Sim-to-real generalization breaks in low-SNR and OOD regimes; the load-bearing sim-to-real caveat. (verify)

  • Media Inhof et al. (DroneRF), 2019; Coluccia et al. (DroneDetect / RFUAV), 2019 to 2025. Single-antenna drone-RF benchmarks. (verify)

  • DARPA RFMLS program, 2017 to 2021; Bihl, Bauer, and Temple, 2017; “No Radio Left Behind,” 2019. Fingerprinting construct and leave-one-unit-out motivation. (verify)

  • O’Shea, Roy, and Clancy, IEEE J-STSP 2018; DeepSig RadioML errata note; Hanna and Hussain, arXiv:2605.27673, 2026. AMC saturation and continuity-column exclusion. (verify)

  • 3GPP TR 38.843, “Study on AI/ML for NR.” Beam, CSI, and positioning use cases without dataset or baseline. (verify revision-year)

  • Le Roux et al., “SDR: Half-baked or Well Done?,” ICASSP 2019, arXiv:1811.02508. SI-SDR for separation. (verify)

  • Davis and Goadrich, ICML 2006; Brodersen et al., ICPR 2010. AUC / AUROC and balanced-accuracy grounding. (verify)

See Also