Tasks validation

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

Scope: the six-lens validation of the 15 task contracts tabulated in Task reference, with depth concentrated on the four v0.1 scored tasks (E-LOC-AOA, E-LOC-LOS, E-ID-DRONE, E-ID-FP) and the E-ID-AMC continuity column. Later-version tasks carry construct and literature coverage only; their empirical and methodology sections land when their recipes pin.

1. Purpose and construct

EMMA (Electromagnetic Multi-task Model Assessment) scores one frozen backbone across five task pillars on raw multi-antenna I/Q (in-phase and quadrature). Construct validity asks, per task, whether the label-plus-metric proxy measures the named property at the abstraction its target domain needs, and whether that domain actually depends on the capability. For each v0.1 task the construct is recorded as: the claim, the proxy, the known omissions, the downstream risk, and the domain utility.

E-LOC-AOA - angle-of-arrival / direction-of-arrival

  • Claim. The backbone can estimate a signal’s direction from multi-antenna phase, and that ability transfers across channel environments.

  • Proxy. Continuous regression of the scalar arrival azimuth aoa_deg (elevation extends the target at v1) in degrees, scored as mean angular error under LEAVE_ONE_ENVIRONMENT_OUT. The label is the rfgen per-emitter metadata at emitters[].aoa_deg.

  • Omissions. Elevation ships at v1; multi-source (multi-emitter) AoA is out of scope at v0.1; near-field geometry is not modeled.

  • Downstream risk. A model that learned environment-specific array calibration, not direction-finding, scores well in-distribution and fails on the held-out environment.

  • Domain utility. Communications positioning, radar direction-finding, and RF (radio-frequency) drone handling all consume an angle estimate. No public multi-antenna dataset scores this today.

E-LOC-LOS - LoS / NLoS (line-of-sight / non-line-of-sight) classification

  • Claim. The backbone separates line-of-sight from non-line-of-sight paths and the decision transfers across environments and frequency bands.

  • Proxy. BalancedAccuracy and AUC (area under the ROC curve) under LEAVE_ONE_ENVIRONMENT_OUT, with a secondary LEAVE_ONE_BAND_OUT axis. The label is the rfgen ray tracer’s path-class flag.

  • Omissions. Only a binary path class; no multi-path-tap accounting.

  • Downstream risk. Class imbalance across environments inflates plain accuracy; balanced accuracy is the mitigation.

  • Domain utility. 5G / 6G positioning and beam management gate on LoS / NLoS state; the 3GPP AI/ML study names it as a use case.

E-ID-DRONE - multi-antenna RF drone detection

  • Claim. The backbone detects drone emitters and the detection transfers across environments and unseen drone models.

  • Proxy. AUC under LEAVE_ONE_ENVIRONMENT_OUT, with a secondary DRONE_MODEL (unseen drone model) holdout. Labels come from the rfgen emitters[].drone_present flag.

  • Omissions. No real-capture drone signatures at v0.1 (synthetic only); real-capture OOD (out-of-distribution) subset lands with the adapter set.

  • Downstream risk. A model memorizes a per-band drone signature rather than learning a transferable feature.

  • Domain utility. Defense and spectrum monitoring both depend on drone detection; every public benchmark (DroneRF, DroneDetect / RFUAV) is single-antenna.

E-ID-FP - emitter / device fingerprinting (SEI: specific emitter identification)

  • Claim. The backbone recognizes an unseen device, not a capture session.

  • Proxy. Top1Accuracy and EER (equal error rate) under LEAVE_ONE_UNIT_OUT, which holds out every capture of one device.

  • Omissions. Receiver front-end and antenna port are not independently randomized at v0.1; the recipe pins a single array.

  • Downstream risk. If leave-one-unit-out is imperfect (a session leaks across the train or eval fold), the score measures session memorization, not device identity.

  • Domain utility. The DARPA RFMLS (RF Machine Learning Systems) lineage and the broader SEI literature depend on exactly this transfer.

E-ID-AMC - automatic modulation classification (continuity)

  • Claim. None at headline level. The column exists so the AMC community can map from RadioML.

  • Proxy. PlainAccuracy under the SNR_REGIME axis (unseen SNR regime). Excluded from the aggregated OOD-avg.

  • Omissions. Saturated, single-antenna-solvable, and its publisher states the data has known errata and is not used in its products; not a generalization claim.

  • Downstream risk. Treating AMC gains as benchmark progress re-imports a saturated task EMMA set out to move beyond.

  • Domain utility. Continuity only; no community depends on EMMA for AMC.

Later-version tasks (construct summary)

  • E-LOC-POS (positioning) extends AoA to a 2D / 3D fix; mean localization error in meters shares AoA’s phase-coherence dependency. E-ID-RWAVE and E-ID-UAVDOP (radar waveform and UAV, unmanned aerial vehicle, micro-Doppler) serve defense emitter identification and carry the same holdout-identity risk as fingerprinting. E-CH-BEAM and E-CH-CSI (CSI: channel state information) serve 3GPP AI/ML; CSI is a deliberate bridge, not a head-to-head feedback competition. E-SC-* (scene understanding) and E-S2T-* (signal-to-text) are frontier tasks with no current community doing them at scale, so they cannot carry adoption at launch.

Sharpest construct threats

(Severity: high, architectural) AoA scored as regression rather than binned classification is part of the construct, not a presentation detail. A binned-AoA variant would let a model with a constant angular offset score high class accuracy while being systematically wrong; the continuous regression target is what exposes that bias. Removing regression breaks the construct.

(Severity: high, architectural) Leave-one-unit-out is the entire construct for E-ID-FP. It must isolate device identity from two confounds the field has never fully separated: capture session and receiver front-end. If the rfgen recipe does not randomize the receiver chain across units, the protocol measures channel or session features rather than device identity, and the score is construct-invalid.

2. Mathematical fidelity

The per-metric formula audit (sign, units, normalization) lives in Metrics validation; this section records task-side math only.

  • Angular regression target. E-LOC-AOA targets an angle in degrees. The error must use a circular distance (wraparound at 360 degrees), not a linear absolute error, or angles near the wrap boundary are penalized incorrectly. (Severity: medium) Open: confirm MeanAngularError uses wrapped angular distance, not plain L1.

  • Balanced accuracy under imbalance. E-LOC-LOS and E-ID-DRONE are class-imbalanced across environments; BalancedAccuracy (macro recall) is the mitigation for LoS / NLoS, and AUC is the threshold-free mitigation for drone detection. Plain accuracy is rejected as a primary metric for both. (Severity: low) Covered in Metrics validation.

  • EER operating point. E-ID-FP reports EER; the equal-error-rate operating point must be interpolated, not snapped to the nearest threshold, to avoid discretization artifacts at small class counts. (Severity: low)

  • NMSE units. E-CH-CSI reports NMSE in dB; the normalization must be over the channel-tensor energy, matching the DeepMIMO convention, or cross-study comparison breaks. (Severity: medium, deferred to v1)

  • Captioning and QA metrics. E-S2T-CAP BLEU (Bilingual Evaluation Understudy), METEOR (Metric for Evaluation of Translation with Explicit ORdering), CIDEr (Consensus-based Image Description Evaluation) and E-S2T-QA exact-match / F1 reuse standard NLP definitions; tokenization of RF captions is the open task-side choice. (Severity: low, deferred to v2)

3. Empirical realism

Each v0.1 task depends on a pinned rfgen recipe (the generator is the dataset). The recipe pages hold the exact rfgen commit, emitters, receiver array, channel environment or environments, labeler, sample counts, splits, and seeds.

  • E-LOC-AOA and E-LOC-LOS depend on the EMMA-LOC v0.1 recipe: a uniform rectangular array in two environments (E-URBAN, E-RURAL) with ray-traced arrival azimuth (emitters[].aoa_deg) and LoS / NLoS path class (emitters[].los_flag).

  • E-ID-DRONE depends on the EMMA-DRONE v0.1 recipe: multi-antenna captures of synthetic drone emitters across models and environments.

  • E-ID-FP depends on the EMMA-FP v0.1 recipe: per-unit emitter instances with a session-stratified leave-one-unit-out split.

  • E-ID-AMC depends on the EMMA-AMC v0.1 recipe, held to RadioML-comparable modulation set and SNR sweep for continuity.

(Severity: high, open) The load-bearing open risk is that rfgen’s inter-antenna phase coherence is not yet independently verified. AoA and beam prediction are unevaluable without correct phase across array elements: if the synthetic array’s per-element phase is not physically consistent, a model can score well on a phase artifact rather than a true direction. This is the single largest empirical-realism threat to the flagship. The proposed verification is a side-by-side against a classical MUSIC / ESPRIT estimator on identical arrays and snapshots (section 5). Cross-link: Datasets validation.

(Severity: medium) Real-capture OOD subsets (Colosseum, POWDER, COSMOS) are not wired at v0.1, so sim-to-real generalization is asserted by recipe design, not yet measured. The RF-Analyzer result that sim-to-real generalization breaks in low-SNR and OOD regimes is the load-bearing caveat here.

4. Literature grounding

The canonical source per task family, drawn from the master reading list:

  • AoA / DoA. Schmidt (MUSIC) and Roy and Kailath (ESPRIT) are the classical array-signal direction-finding anchors; both ground that AoA is a continuous angular quantity. Van Trees is the textbook of record.

  • AMC. O’Shea et al. (Convolutional Radio Modulation Recognition Networks; RadioML 2018), the DeepSig datasets-page note that RML2016 has known errata and is not used in DeepSig products, and Hanna and Hussain (matched hyperparameter search collapses apparent gaps) ground the saturation and continuity-column exclusion.

  • Fingerprinting / SEI. The DARPA RFMLS program, Bihl, Bauer, and Temple (feature representation), and “No Radio Left Behind” (cross-session device identification) ground the leave-one-unit-out construct.

  • Beam and CSI. 3GPP TR 38.843 names the AI/ML use cases with no dataset or baseline; Alkhateeb (DeepMIMO) and Djordjevic, Ali, and Alkhateeb (LWM) are the CSI camp the bridge targets.

  • Drone and UAV. Media Inhof et al. (DroneRF) and Coluccia et al. (DroneDetect / RFUAV) are the single-antenna drone-RF benchmarks EMMA moves beyond.

  • Separation. Le Roux et al. (SI-SDR) define the separation metric.

  • Captioning and QA. Papineni (BLEU), Banerjee and Lavie (METEOR), Vedantam (CIDEr), and Rajpurkar (SQuAD exact-match and F1) define the signal-to-text metrics.

  • AUC / AUROC and balanced accuracy. Davis and Goadrich, and Brodersen et al., ground the threshold-free and imbalance-robust classification metrics.

  • Benchmark methodology. SUPERB (one backbone, many readouts, one aggregate) and WILDS (distribution-shift-first, leave-one-X-out) are the structural precedents.

(Severity: low) Several citations carry (verify) in the reading list; none are presented as fact here until verified. No new literature entries are required for this report.

5. Experimental methodology and planned tests

Tests are written so an implementer can port them directly to tests/validation/tasks/ once src/emma/ lands. Each entry records the falsifiable claim, the failure mode, the design, the sample size, the statistical test, the tolerance, and the gold-standard reference.

E-LOC-AOA - tests/validation/tasks/test_loc_aoa.py

  • Falsifiable claim. A backbone that learned direction-finding transfers across environments; its mean angular error on the held-out environment is better than the per-task chance baseline and within a bounded gap of a classical estimator.

  • Failure mode. The model predicts the mean training angle, or overfits training-environment multipath and collapses on the held-out environment.

  • Design. LEAVE_ONE_ENVIRONMENT_OUT over E-URBAN and E-RURAL; report wrapped mean angular error per environment and the aggregate.

  • Sample size. Minimum 2000 scenes per environment per split (proposed; pinned in the recipe).

  • Statistical test. Paired bootstrap 95% confidence interval on per-scene angular error, and a one-sided test that the model beats the mean-angle predictor.

  • Tolerance. Regression mean angular error within a fixed delta of the MUSIC gold standard on identical arrays and snapshots (delta pinned with the first data release).

  • Gold-standard reference. Classical MUSIC and ESPRIT estimators on the same array geometry and snapshots, computed via scipy / Sionna primitives.

E-LOC-LOS - tests/validation/tasks/test_loc_los.py

  • Falsifiable claim. LoS / NLoS detection transfers across environments and bands above the class-prior baseline.

  • Failure mode. The model exploits an environment-specific path-loss threshold; balanced accuracy drops on the held-out environment.

  • Design. LEAVE_ONE_ENVIRONMENT_OUT with a layered LEAVE_ONE_BAND_OUT fold; report balanced accuracy and AUC.

  • Sample size. Minimum 2000 scenes per environment, balanced across LoS / NLoS where the recipe allows.

  • Statistical test. McNemar paired test on per-scene correctness against the prior-baseline classifier; bootstrap CI on AUC.

  • Tolerance. Balanced accuracy beats the majority-class baseline by a pinned margin.

  • Gold-standard reference. The rfgen ray tracer’s ground-truth path-class label.

E-ID-DRONE - tests/validation/tasks/test_id_drone.py

  • Falsifiable claim. Drone detection transfers across environments and unseen drone models above the prior baseline.

  • Failure mode. The model memorizes a per-band drone signature; AUC collapses on the held-out drone model.

  • Design. LEAVE_ONE_ENVIRONMENT_OUT with a secondary DRONE_MODEL fold; report AUC.

  • Sample size. Minimum 1000 scenes per drone model per environment.

  • Statistical test. DeLong test on AUC differences.

  • Tolerance. AUC beats the prior baseline by a pinned margin on the unseen drone model.

  • Gold-standard reference. rfgen drone-emitter class labels; eventually the DroneRF-style real-capture subset via the adapter.

E-ID-FP - tests/validation/tasks/test_id_fp.py

  • Falsifiable claim. Device recognition transfers to an unseen unit under leave-one-unit-out.

  • Failure mode. The model exploits capture-session leakage or receiver front-end artifacts if the holdout is imperfect.

  • Design. LEAVE_ONE_UNIT_OUT with session-stratified folds; report Top-1 and EER.

  • Sample size. Minimum 20 units, multiple sessions per unit.

  • Statistical test. Paired bootstrap CI on Top-1 across held-out units; CI on the EER operating point.

  • Tolerance. Top-1 beats the closed-set chance baseline by a pinned margin; EER below a pinned ceiling.

  • Gold-standard reference. Per-unit emitter labels with a documented session and receiver-chain randomization (the open recipe question in section 1).

E-ID-AMC - tests/validation/tasks/test_id_amc.py

  • Falsifiable claim. None at headline level; the test only confirms continuity (the column runs and reports a number comparable to RadioML).

  • Failure mode. The column silently re-enters the aggregate; the test guards the exclusion.

  • Design. SNR_REGIME (unseen-SNR-regime) holdout; report plain accuracy.

  • Gold-standard reference. RadioML 2018.01A class set for mapping.

Open methodology question

(Severity: high, architectural, deferred to open questions) Leave-one-environment-out with only two environments (E-URBAN, E-RURAL) gives \(N = 2\) for the environment-level statistic. A two-point mean has essentially no power to distinguish models at the environment stratum, and a single outlier environment dominates the aggregate. The proposed mitigations are: (a) add a third environment as soon as the recipe allows, raising \(N\) to 3; (b) report per-environment scores alongside the aggregate, never the aggregate alone; and © treat the v0.1 aggregate as comparative ranking under a fixed protocol, not a population estimate of transfer. This is the sharpest methodology threat to the v0.1 leaderboard and is named explicitly rather than hidden.

6. Robustness boundaries

Each boundary is a claim about where validity breaks, paired with a proposed probe.

  • Metric gaming: AoA predicts the mean angle. (Severity: high) A model that outputs the training-mean angle scores nonzero on a binned metric and near-zero only under a regression metric that penalizes systematic offset. Probe: include a mean-angle predictor as a floor baseline in test_loc_aoa.py; a real model must beat it one-sided. The regression metric is the mitigation.

  • Metric gaming: fingerprint model exploits capture-session leakage. (Severity: high) If leave-one-unit-out leaks a session across train or eval, a model can score high on session features. Probe: a session-confound audit that re-splits with strict session isolation and checks whether Top-1 moves; a large move flags leakage. See test_id_fp.py.

  • Small-N environment statistics. (Severity: high) Covered in section 5; the two-environment aggregate is fragile. Probe: a leave-one-environment-out jackknife (drop each environment in turn) to bound the aggregate’s sensitivity to any single environment.

  • Sim-to-real confound. (Severity: medium) Synthetic-only v0.1 tasks assert realism by recipe design. Probe: once the real-capture adapters land, run a sim-to-real gap probe (tests/validation/tasks/test_sim_to_real.py, proposed) comparing synthetic-trained scores to real-capture scores; the RF-Analyzer low-SNR break is the expected failure region.

  • Phase-coherence fidelity. (Severity: high) The flagship depends on physically consistent inter-antenna phase. Probe: the MUSIC / ESPRIT gold-standard side-by-side in test_loc_aoa.py doubles as a phase-coherence check; a classical estimator failing to recover known directions on rfgen scenes flags a generation bug, not a model bug.

  • Drone-model holdout leakage. (Severity: medium) The secondary DRONE_MODEL axis must not share a waveform or band artifact with training models. Probe: a band-stratified re-split in test_id_drone.py.

7. Synthesis

Load-bearing claims supported. The five-pillar structure maps to named domain buyers (positioning, defense, spectrum, radar, communications), and the flagship (AoA regression under environment transfer) is construct-valid and unclaimed by existing benchmarks. The continuity-column exclusion of AMC is literature-grounded. The leave-one-unit-out construct for fingerprinting is the right protocol, conditional on recipe rigor.

Gaps surfaced. (1) rfgen inter-antenna phase coherence is unverified and is the single largest threat to the flagship. (2) Leave-one-environment-out at \(N = 2\) environments is underpowered for an environment-level statistic. (3) Real-capture OOD is absent at v0.1, so sim-to-real realism is asserted, not measured. (4) The E-ID-FP receiver-chain randomization is not pinned, leaving the session-confound door open.

Recommended actions. (1) Land the MUSIC / ESPRIT gold-standard side-by-side as the first tests/validation/tasks/ test; it validates both the metric and the generator. (2) Pin the E-ID-FP receiver-chain and session randomization in the recipe before the task can be scored. (3) Add a third environment to the v0.1 recipe if feasible; otherwise report per-environment scores alongside every aggregate. (4) Wire the real-capture adapters and the sim-to-real probe before claiming empirical realism.

Top three findings.

  1. (high) rfgen phase-coherence fidelity is unverified and load-bearing for AoA and beam; the MUSIC / ESPRIT gold-standard test is the proposed first verification.

  2. (high) Two-environment leave-one-environment-out is underpowered; the v0.1 aggregate is a comparative ranking, not a population estimate, and must ship with per-environment scores.

  3. (high) E-ID-FP construct validity is entirely conditional on a rigorous leave-one-unit-out recipe that isolates device identity from session and receiver-chain confounds; the recipe must pin that isolation before scoring.

References

  • Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE Trans. Acoustics, Speech, Signal Processing 1986, DOI:10.1109/TASSP.1986.1164830. MUSIC; the continuous-AoA construct and the gold-standard estimator. (verify)

  • Roy and Kailath, “ESPRIT: Estimation of signal parameters via rotational invariance techniques,” IEEE Trans. Acoustics, Speech, Signal Processing 1989, DOI:10.1109/29.32276. Second classical estimator for the phase-coherence probe. (verify)

  • Van Trees, “Optimum Array Processing,” Wiley 2002. Array-processing textbook of record. (verify)

  • O’Shea, Roy, and Clancy, “Over-the-Air Deep Learning Based Radio Signal Classification,” IEEE J-STSP 2018. RadioML lineage. (verify)

  • DeepSig, “RadioML datasets,” datasets page (deepsig.ai/datasets). The publisher’s public note that RML2016 has known errata and is not used in DeepSig products; grounds the E-ID-AMC continuity-column exclusion. (verify URL)

  • Hanna and Hussain, “Robust Low-SNR Modulation Classification,” 2026, arXiv:2605.27673. AMC saturation. (verify)

  • DARPA RFMLS program, 2017 to 2021. Fingerprinting capability-without-benchmark precedent. (verify program record)

  • Bihl, Bauer, and Temple, “Feature representation for RF fingerprinting,” 2017. SEI literature. (verify)

  • “No Radio Left Behind,” 2019. Cross-session device identification; leave-one-unit-out motivation. (verify)

  • 3GPP TR 38.843, “Study on AI/ML for NR.” Beam and CSI use cases without dataset or baseline. (verify revision-year)

  • Alkhateeb, “DeepMIMO,” 2019, arXiv:1902.06435; Djordjevic, Ali, and Alkhateeb, “LWM,” 2024, arXiv:2411.08872. CSI camp. (verify)

  • Media Inhof et al. (DroneRF), 2019; Coluccia et al. (DroneDetect / RFUAV), 2019 to 2025. Single-antenna drone-RF benchmarks. (verify)

  • Le Roux, Wisdom, Erdogan, and Hershey, “SDR: Half-baked or Well Done?,” ICASSP 2019, arXiv:1811.02508. SI-SDR. (verify)

  • Papineni et al., “BLEU,” ACL 2002; Banerjee and Lavie, “METEOR,” ACL 2005 (verify); Vedantam et al., “CIDEr,” CVPR 2015 (verify); Rajpurkar et al., “SQuAD,” EMNLP 2016, arXiv:1606.05250 (verify). Signal-to-text metrics.

  • Davis and Goadrich, “The relationship between Precision-Recall and ROC curves,” ICML 2006. AUC / AUROC. (verify)

  • Brodersen et al., “The balanced accuracy and its posterior distribution,” ICPR 2010. Balanced accuracy. (verify)

  • Yang et al., “SUPERB,” Interspeech 2021, arXiv:2105.01051 (verify); Koh et al., “WILDS,” ICML 2021, arXiv:2012.07421. Benchmark methodology precedents.

  • “RF-Analyzer,” 2026, arXiv:2605.04676. Sim-to-real generalization breaks in low-SNR and OOD regimes. (verify)

See Also