Metrics validation¶
Warning
Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.
Scope: the six-lens validation of the metric contracts in Metric reference and the two algorithm contracts in Aggregation and Sim-to-real gap, with depth concentrated on the mathematical-fidelity lens. The five v0.1 metrics (MeanAngularError, BalancedAccuracy, AUC, Top1Accuracy with EER, PlainAccuracy) carry full audits, alongside the aggregator (OODAvg) and the standalone fidelity-gap reporter (SimToRealGap, not an Aggregator because its pairwise (synth_score, real_score) signature is incompatible with the fold-reduction contract). Later-version metrics (NMSE, SISDR, AUROC, TopKAccuracy, captioning, QA) carry construct and literature coverage; their empirical and methodology sections land when their recipes pin.
1. Purpose and construct¶
The metric layer’s claim is narrow and load-bearing: each formula is the correct proxy for the property its task names, at the abstraction the target domain needs, with the right sign, units, and normalization. Construct validity at this layer asks three questions per metric. Is the formula the standard definition or a documented custom one? Does the direction match the property (an error is lower-is-better, an accuracy is higher-is-better)? Is the metric chosen to resist the gaming mode its task is vulnerable to?
E-LOC-AOA/ MAE (mean angular error). The construct is continuous regression on a circular quantity. The proxy is the mean wrapped angular error. The gaming mode is a binned-AoA (angle of arrival) metric that hides systematic angular bias behind class accuracy; the regression target is the mitigation, and the circular distance handles wraparound.E-LOC-LOS/ balanced accuracy. The construct is LoS / NLoS (line-of-sight / non-line-of-sight) detection under class imbalance. The proxy is macro recall. The gaming mode is plain accuracy inflated by a majority path class; balanced accuracy is the mitigation.E-ID-DRONE,E-SC-SENSE/ AUC. The construct is threshold-free binary detection. The proxy is the area under the ROC (receiver operating characteristic) curve. The gaming mode is a threshold-tuned accuracy; AUC is threshold-free.E-ID-FP/ top-1 and EER. The construct is closed-set device recognition plus open-set verification. The proxy is top-1 accuracy and the equal-error-rate operating point. The gaming mode is a snapped (non-interpolated) EER with discretization artifacts; interpolation is the mitigation.E-ID-AMC/ plain accuracy. The construct is continuity only (no headline claim). The proxy is multiclass accuracy. The gaming mode is treating AMC (automatic modulation classification) gains as benchmark progress; the aggregate exclusion is the mitigation.
2. Mathematical fidelity¶
The line-by-line audit. For each metric: the formula, the sign, the units, the normalization, and the cited definition it is checked against. The full formulas live in Metric reference; this section records the fidelity verdict.
Mean angular error (MeanAngularError)¶
Formula. \(\mathrm{MAE} = \frac{1}{N}\sum_i d_{\mathrm{circ}}(\hat{\theta}_i, \theta_i)\) with circular distance \(d_{\mathrm{circ}} = \min(\delta, 360 - \delta)\), \(\delta = (|\hat{\theta} - \theta| \bmod 360)\).
Sign. Lower is better. Correct: an error quantity.
Units. Degrees. Correct: AoA is an angle.
Normalization. None at the metric level; the raw error is normalized to \([0,1]\) only at aggregation by frozen anchors (see Aggregation).
Cited definition. Schmidt (MUSIC) and Roy and Kailath (ESPRIT) ground AoA as a continuous angular quantity; the circular distance is the standard distance on the circle. (verify)
Verdict. Fidelity passes conditional on the circular distance being implemented, not plain L1 (linear absolute error). (Severity: medium, open) Confirm the implementation uses \(d_{\mathrm{circ}}\), because a plain \(|\hat{\theta} - \theta|\) penalizes angles near the wrap boundary incorrectly (359 degrees vs 1 degree costs 358 instead of 2). This is the single load-bearing math check for the flagship.
Balanced accuracy (BalancedAccuracy)¶
Formula. \(\mathrm{BalAcc} = \frac{1}{C}\sum_c \frac{TP_c}{TP_c + FN_c}\), the macro recall.
Sign. Higher is better. Correct.
Units. Dimensionless, \([0,1]\).
Normalization. None beyond the recall average; aggregated by frozen anchors.
Cited definition. Brodersen et al., “The balanced accuracy and its posterior distribution,” ICPR 2010. (verify)
Verdict. Fidelity passes. Matches
torchmetrics.classification.MulticlassAccuracy(average="macro"). Plain accuracy is rejected for this task.
AUC and AUROC (AUC, AUROC)¶
Formula. \(\mathrm{AUC} = \int_0^1 \mathrm{TPR}(t)\,d\mathrm{FPR}(t) = P(\hat{s}_+ > \hat{s}_-)\).
Sign. Higher is better. Correct.
Units. Dimensionless, \([0,1]\).
Normalization. None; threshold-free.
Cited definition. Davis and Goadrich, “The relationship between Precision-Recall and ROC curves,” ICML 2006. (verify)
Verdict. Fidelity passes. The threshold-free property is why AUC is chosen over threshold-tuned accuracy for
E-ID-DRONEandE-SC-SENSE. AUC exposes anum_classesconstructor parameter that selectstorchmetrics.classification.BinaryAUROCfornum_classes=2(E-SC-SENSE) andtorchmetrics.classification.MulticlassAUROCfornum_classes > 2(E-ID-DRONE); AUROC is strictly binary forE-SC-ANOM.
Top-1 accuracy and EER (Top1Accuracy, EER)¶
Formula. \(\mathrm{Top\text{-}1} = \frac{1}{N}\sum_i \mathbf{1}[\arg\max_c \hat{p}_i = y_i]\); \(\mathrm{EER}: \mathrm{FAR}(\tau^\*) = \mathrm{FRR}(\tau^\*)\).
Sign. Top-1 higher is better; EER lower is better. Correct.
Units. Dimensionless, \([0,1]\).
Normalization. None; aggregated by frozen anchors.
Cited definition. Standard classification accuracy; the equal-error-rate operating point from the biometric-verification literature. The EER is custom on
torch, built by interpolating the crossing point on thetorchmetrics.classification.BinaryROCcurve because torchmetrics ships no direct EER primitive at the time of writing. (verify SEI grounding)Verdict. Fidelity passes conditional on EER interpolation. (Severity: low) The EER crossing \(\tau^\*\) must be interpolated on the ROC curve, not snapped to the nearest evaluated threshold, to avoid discretization artifacts at small class counts.
Plain accuracy (PlainAccuracy)¶
Formula. \(\mathrm{Acc} = \frac{1}{N}\sum_i \mathbf{1}[\arg\max_c \hat{p}_i = y_i]\).
Sign. Higher is better. Correct.
Units. Dimensionless, \([0,1]\).
Normalization. None; excluded from OOD-avg (out-of-distribution average).
Cited definition. Standard multiclass accuracy.
Verdict. Fidelity passes. The construct caveat is that this metric is continuity only for
E-ID-AMCand is excluded from the aggregate.
NMSE (NMSE, deferred to v1)¶
Formula. \(\mathrm{NMSE} = 10\log_{10}\bigl(\sum_i\|\hat{\mathbf{H}}_i - \mathbf{H}_i\|_F^2 / \sum_i\|\mathbf{H}_i\|_F^2\bigr)\) in dB.
Sign. Lower is better. Correct.
Units. Decibels (dB). (Severity: medium, deferred to v1) Confirm the normalization is over channel-tensor energy, matching the DeepMIMO convention, or cross-study comparison breaks.
SI-SDR (SISDR, deferred to v1)¶
Formula. \(\mathrm{SI\text{-}SDR} = 10\log_{10}(\|s_{\mathrm{target}}\|^2 / \|e_{\mathrm{noise}}\|^2)\) with optimal scale \(\alpha = \langle\hat{s},s\rangle/\|s\|^2\).
Sign. Higher is better. Correct.
Units. Decibels (dB).
Cited definition. Le Roux et al., “SDR: Half-baked or Well Done?,” ICASSP 2019. (verify)
Verdict. Fidelity passes conditional on the zero-reference guard: the projection \(\alpha\) is undefined when \(\|s\|^2 = 0\), so silent references must be excluded from the reduction.
OOD-avg (OODAvg)¶
OODAvg is the sole Aggregator subclass. SimToRealGap (below) is a standalone reporter, not an Aggregator.
Formula. Piecewise normalization to \([0,1]\) with frozen anchors \((m_{\min}, m_{\max})\), lower-is-better inverted first, then the mean over the four aggregated tasks.
Sign. Higher is better. Correct.
Units. Dimensionless, \([0,1]\).
Normalization. The central contract of this metric. Two fidelity checks. (1) Inversion branch. Lower-is-better metrics use \(\hat{m} = (m_{\max} - m)/(m_{\max} - m_{\min})\), so
E-LOC-AOA(an error) is correctly inverted before averaging. (2) Anchor freezing. Anchors are fixed per release and never recomputed per submission; this is the anti-gaming property.Verdict. Fidelity passes. The aggregate is mathematically sound conditional on the anchors being published and frozen at release.
Sim-to-real gap (SimToRealGap)¶
SimToRealGap is a standalone fidelity-gap reporter, not an Aggregator: its pairwise (synth_score, real_score) call signature is not substitutable for the __call__(fold_scores) fold-reduction contract, so subclassing Aggregator would be a Liskov violation.
Formula. \(\Delta_{\text{s2r}} = 100 \cdot (\hat{m}_{\text{synth}} - \hat{m}_{\text{real}})\).
Sign. Positive gap means the model scored higher on synthetic than on real captures (overfit to rfgen). Correct convention.
Units. Percentage points (pp). Correct: the \(100\,\cdot\) scaling maps the \([0,1]\) difference to pp.
Normalization. Reuses the OOD-avg normalization on both splits, so the two scores are comparable.
Verdict. Fidelity passes. The gap is reported alongside OOD-avg, never inside it.
3. Empirical realism¶
The metric layer’s empirical claim is not about the data (that is the datasets layer) but about the metric implementation: a known model on a controlled scene must produce a sane value. The proposed tests are golden-value checks, not recipe realism checks. They land as proposed pytest plans (section 5) that skip-with-rationale until src/emma/ exists; literature evidence for the sanity bounds is cited in section 4.
Perfect prediction. A model that predicts the true label exactly must yield MAE of 0, balanced accuracy of 1, AUC of 1, top-1 of 1, and OOD-avg of 1. This is the upper-bound sanity check.
Chance prediction. A model that predicts a fixed prior must yield AUC near 0.5, balanced accuracy near \(1/C\) for \(C\) balanced classes, and EER near 0.5. This is the lower-bound sanity check.
Mean-angle predictor. A model that outputs the training-mean angle on
E-LOC-AOAmust score worse than any real model under the regression metric; it is the documented floor baseline.Controlled angular offset. A model that predicts the true angle plus a constant offset must incur a mean error equal to that offset (modulo wraparound), confirming the circular distance.
Realism of the underlying scenes (rfgen phase coherence, real-capture fidelity) is audited in Datasets validation and Tasks validation; this report depends on those, it does not duplicate them.
4. Literature grounding¶
The canonical source per metric, drawn from the master reading list, and the library reuse check.
MAE. Schmidt (MUSIC) and Roy and Kailath (ESPRIT) ground AoA as continuous. Custom on
torchbecausetorchmetricsships no circular-distance regression metric; the custom wrap is minimal and documented. (verify)Balanced accuracy. Brodersen et al. Reuses
torchmetrics. (verify)AUC / AUROC. Davis and Goadrich. Reuses
torchmetrics. (verify)Top-1 / EER. Top-1 reuses
torchmetrics.classification.MulticlassAccuracy. EER is custom ontorch, interpolating the crossing point on atorchmetrics.classification.BinaryROCcurve because torchmetrics ships no direct EER primitive. (verify SEI grounding)Plain accuracy. Standard. Reuses
torchmetrics.Top-k accuracy. Standard. Reuses
torchmetrics.NMSE. DeepMIMO convention. Custom on
torchbecausetorchmetricsships no channel-tensor NMSE. (verify)SI-SDR. Le Roux et al. Reuses
torchmetrics.audio.ScaleInvariantSignalDistortionRatio. (verify)Captioning. Papineni (BLEU, Bilingual Evaluation Understudy), Banerjee and Lavie (METEOR, Metric for Evaluation of Translation with Explicit ORdering), Vedantam (CIDEr-D, Consensus-based Image Description Evaluation). BLEU reuses
torchmetrics.text.BLEUScore; METEOR reusesnltk.translate.meteor_score; CIDEr-D targets the canonical Vedantam variant viapycocoevalcapuntil atorchmetricsequivalent ships. (verify METEOR, CIDEr-D)Exact match / F1. Rajpurkar (SQuAD). Reuses
torchmetrics. (verify)OOD-avg. SUPERB (one aggregate) and WILDS (generalization-first) precedents. Custom aggregator; the normalization math is EMMA’s own, documented in Aggregation. (verify SUPERB)
Sim-to-real gap. RF-Analyzer (the sim-to-real caveat). Custom on
torch; the gap is a percentage-point difference of two normalized scores. (verify)
Library-first check¶
Each metric maps to a torchmetrics, scipy, or numpy function where one exists. The three custom-on-torch cases (MeanAngularError, NMSE, SimToRealGap) each carry a minimal custom rationale: no library covers the RF-specific circular distance, the channel-tensor energy normalization, or the paired normalized-score difference. EER is a fourth custom case in a weaker sense, interpolating the FAR-equals-FRR crossing on a torchmetrics.classification.BinaryROC curve because torchmetrics ships no direct EER primitive. Any further custom logic is a finding, even if functionally correct.
(Severity: low) Several citations carry (verify) in the reading list; none are presented as fact here until verified. No new literature entries are required for this report.
5. Experimental methodology and planned tests¶
Tests are written so an implementer can port them directly to tests/validation/metrics/ once src/emma/ lands. Each records the falsifiable claim, the failure mode, the design, the gold-standard reference, and the tolerance. Until src/emma/ exists, every test in this section is gated to skip-with-rationale rather than collect-as-failure; the active lens emits the plan plus the literature evidence, not a green check.
Golden-value tests - tests/validation/metrics/test_golden_values.py¶
Falsifiable claim. A perfect prediction yields the metric ceiling; a chance prediction yields the metric floor.
Failure mode. The implementation has a sign error, a unit error, or a wrong reduction.
Design. Construct synthetic predictions and labels where the truth is known by construction. Assert MAE of 0, balanced accuracy of 1, AUC of 1, top-1 of 1, and OOD-avg of 1 for a perfect model; assert AUC near 0.5 and balanced accuracy near \(1/C\) for a chance model.
Gold-standard reference. The known-by-construction labels.
Tolerance. Exact for integer-reducible cases; floating-point tolerance (\(10^{-6}\)) for the rest.
Mean angular error - tests/validation/metrics/test_mean_angular_error.py¶
Falsifiable claim. The metric uses the circular distance, not plain L1.
Failure mode. Wraparound is mishandled; 359 degrees vs 1 degree costs 358 instead of 2.
Design. Feed a prediction and truth that straddle the wrap boundary; assert the error equals the circular distance. Feed a constant-offset model; assert the mean error equals the offset (modulo wraparound).
Gold-standard reference. The circular-distance formula.
Tolerance. Exact.
Balanced accuracy and AUC - tests/validation/metrics/test_balanced_accuracy.py, test_auc.py¶
Falsifiable claim. Balanced accuracy is macro recall and resists class imbalance; AUC is threshold-free.
Failure mode. The implementation reports plain accuracy under imbalance, or AUC depends on the threshold.
Design. Construct an imbalanced binary set; confirm balanced accuracy differs from plain accuracy. Confirm AUC is invariant to monotone score rescaling (threshold-free).
Gold-standard reference.
torchmetrics.classification.MulticlassAccuracy(average="macro")andBinaryAUROC.Tolerance. Floating-point tolerance against the library reference.
EER - tests/validation/metrics/test_eer.py¶
Falsifiable claim. The EER is interpolated, not snapped to the nearest threshold.
Failure mode. Discretization artifacts at small class counts.
Design. Construct a score set where the FAR and FRR curves cross between evaluated thresholds; assert the reported EER lies on the interpolation, not at a grid point.
Gold-standard reference. Linear interpolation on the
torchmetrics.classification.BinaryROCcurve at the FAR-equals-FRR crossing; torchmetrics ships no direct EER primitive to compare against.Tolerance. Floating-point tolerance.
OOD-avg - tests/validation/metrics/test_ood_avg.py¶
Falsifiable claim. OODAvg, the sole Aggregator subclass, inverts lower-is-better metrics and averages the four v0.1 tasks.
Failure mode. The inversion branch is missing or wrong;
E-ID-AMCleaks into the aggregate.Design. Feed four known per-task scores (one lower-is-better, three higher-is-better) with known anchors; assert the normalized mean. Feed a fifth
E-ID-AMCscore and confirm it does not change the aggregate.Gold-standard reference. The piecewise formula in Aggregation.
Tolerance. Floating-point tolerance.
Sim-to-real gap - tests/validation/metrics/test_sim_to_real_gap.py¶
Falsifiable claim. The gap is \(100 \cdot (\hat{m}_{\text{synth}} - \hat{m}_{\text{real}})\) in percentage points and is independent of OOD-avg.
Failure mode. The gap is folded into OOD-avg, or the scaling is wrong.
Design. Feed two normalized scores; assert the gap in pp. Confirm changing the real score does not change a separately computed OOD-avg.
Gold-standard reference. The gap formula.
Tolerance. Floating-point tolerance.
Class roles and LSP guard - tests/validation/metrics/test_class_roles.py¶
Falsifiable claim. SimToRealGap is a standalone reporter and does not inherit Aggregator; OODAvg is the sole Aggregator subclass.
Failure mode. A future refactor re-introduces
SimToRealGap(Aggregator), recreating the Liskov violation because the(synth_score, real_score)call signature cannot substitute for__call__(fold_scores).Design.
assert not issubclass(SimToRealGap, Aggregator)andassert issubclass(OODAvg, Aggregator).Gold-standard reference. The class hierarchy contract in Metrics API.
Tolerance. Exact.
6. Robustness boundaries¶
Each boundary is a claim about where the metric validity breaks, paired with a proposed probe. Probes are proposed pytest plans that skip-with-rationale until src/emma/ exists; the boundary itself is a literature-grounded claim that holds regardless of implementation.
Angular wraparound (MAE). (Severity: high) A plain-L1 implementation mishandles angles near the 0 or 360 degree boundary. Probe: the wrap-boundary case in
test_mean_angular_error.py. The circular distance is the mitigation.Class imbalance hiding behind plain accuracy. (Severity: medium) Plain accuracy on an imbalanced split inflates a majority-class baseline. Probe: the imbalanced-set case in
test_balanced_accuracy.py. Balanced accuracy and AUC are the mitigation.Mean-predictor AoA. (Severity: high) A model that outputs the training-mean angle scores nonzero on a binned metric and near-zero only under a regression metric that penalizes systematic offset. Probe: include a mean-angle predictor as a floor baseline; a real model must beat it one-sided. The regression target is the mitigation.
EER discretization. (Severity: low) A snapped EER produces artifacts at small class counts. Probe: the between-threshold crossing case in
test_eer.py. Interpolation is the mitigation.Empty prediction set. (Severity: medium) A metric reduction over zero samples is undefined (division by zero). Probe: feed an empty prediction set and assert the implementation raises a documented error rather than returning NaN (not a number). Guard, do not silently average.
Single environment. (Severity: high, covered in Tasks validation) Leave-one-environment-out at \(N = 2\) environments is underpowered; the metric is correct but the mean is fragile. Probe: a jackknife (drop each environment in turn) to bound the aggregate’s sensitivity.
Zero-reference SI-SDR. (Severity: medium, deferred to v1) A silent reference makes the scale projection \(\alpha\) undefined. Probe: feed a zero-reference sample and assert it is excluded from the reduction rather than producing an infinite score.
Anchor gaming (OOD-avg). (Severity: medium) If anchors were recomputed per submission, a submission could flatter its own scores. Probe: confirm the anchors are read from the frozen release manifest and never from the submission. Frozen anchors are the mitigation.
Sim-to-real gap folded into OOD-avg. (Severity: medium) The gap is a fidelity-gap reporter, not an Aggregator output. Folding it into OOD-avg would let a model compensate for a large gap with strong synthetic scores and hide a fidelity failure. Probe: the independence assertion in
test_sim_to_real_gap.pyand the LSP guard intest_class_roles.py. The standalone-reporter design is the mitigation.
7. Synthesis¶
Load-bearing claims supported. The v0.1 metric set reuses standard library definitions wherever one exists (balanced accuracy, AUC, top-1, plain accuracy), with EER custom on a torchmetrics ROC curve, MAE custom on torch, and the OODAvg and SimToRealGap contracts each carrying a minimal documented rationale. Signs, units, and normalizations are correct against their cited definitions, conditional on the open checks below. The OOD-avg normalization is mathematically sound, and its frozen-anchor property is the right anti-gaming design; OODAvg is the sole Aggregator subclass. The sim-to-real gap is correctly a percentage-point column reported alongside, not inside, the aggregate, and the standalone-reporter class design keeps the Liskov contract structural.
Gaps surfaced. (1) The circular-distance implementation of MAE is the single load-bearing math check for the flagship and is unverified until code lands. (2) The EER interpolation on the torchmetrics ROC curve is unverified. (3) The NMSE channel-tensor normalization convention is deferred to v1. (4) The zero-reference SI-SDR guard is deferred to v1. (5) The frozen-anchor mechanism is asserted by contract, not yet enforced by code. (6) The CIDEr-D target via pycocoevalcap carries (verify) until a torchmetrics equivalent ships.
Recommended actions. (1) Land tests/validation/metrics/test_golden_values.py and test_mean_angular_error.py first; they validate the flagship metric and the circular distance. (2) Pin the EER interpolation in test_eer.py. (3) Add a guard test for the empty-prediction-set edge case across all metrics. (4) Wire the frozen-anchor read from the release manifest into test_ood_avg.py once the manifest exists. (5) Land test_class_roles.py as a structural guard so the SimToRealGap standalone-reporter contract cannot silently regress to an Aggregator subclass.
Top three findings.
(high) MAE must use the circular distance, not plain L1; the wrap-boundary golden test is the proposed first verification and is the single load-bearing math check for
E-LOC-AOA.(high) Plain accuracy is rejected for the class-imbalanced scored tasks; balanced accuracy and AUC are the mitigations, and the imbalanced-set tests enforce the choice.
(medium) OOD-avg anti-gaming depends on anchors being frozen per release and read from the manifest, never recomputed per submission; the anchor-read test enforces this once the manifest lands.
References¶
Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE Trans. Acoustics, Speech, Signal Processing 1986, DOI:10.1109/TASSP.1986.1164830. MUSIC; AoA as a continuous angular quantity. (verify)
Roy and Kailath, “ESPRIT: Estimation of signal parameters via rotational invariance techniques,” IEEE Trans. Acoustics, Speech, Signal Processing 1989, DOI:10.1109/29.32276. (verify)
Le Roux, Wisdom, Erdogan, and Hershey, “SDR: Half-baked or Well Done?,” ICASSP 2019, arXiv:1811.02508. SI-SDR. (verify)
Papineni et al., “BLEU,” ACL 2002; Banerjee and Lavie, “METEOR,” ACL 2005 (verify); Vedantam et al., “CIDEr,” CVPR 2015 (verify); Rajpurkar et al., “SQuAD,” EMNLP 2016, arXiv:1606.05250 (verify). Captioning and QA metrics.
Davis and Goadrich, “The relationship between Precision-Recall and ROC curves,” ICML 2006. AUC and AUROC. (verify)
Brodersen et al., “The balanced accuracy and its posterior distribution,” ICPR 2010. Balanced accuracy. (verify)
Yang et al., “SUPERB,” Interspeech 2021, arXiv:2105.01051 (verify); Koh et al., “WILDS,” ICML 2021, arXiv:2012.07421. Aggregation precedents.
“RF-Analyzer,” 2026, arXiv:2605.04676. Sim-to-real caveat. (verify)
torchmetrics >= 0.11;nltk;torch >= 2.0. Library function paths per Metric reference.
See Also¶
Metric reference: the contracts this report validates.
Aggregation: the OOD-avg normalization contract.
Sim-to-real gap: the gap contract.
Tasks validation: the task-side math and gaming probes cross-referenced in section 2.
Validation methodology: the six-lens framework and report shape.
Literature: the master reading list these citations are drawn from.