Metric reference¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

Every EMMA (Electromagnetic Multi-task Model Assessment) metric is a standard metric reused from torchmetrics, scipy, or numpy wherever a library already defines it. A metric is flagged custom only when no library implementation covers the RF (radio-frequency)-specific quantity; the three custom cases (MeanAngularError, NMSE, and SimToRealGap) wrap a minimal computation on torch tensors with a documented rationale, never a reimplementation of an existing library metric. The table below is the normative per-metric contract; the per-metric subsections give the formula, units, and edge cases. Which metric each task uses is fixed in Task reference.

Metric table¶

Direction is the optimization sense: higher means higher is better; lower means lower is better. The Library column names the reused function; custom means EMMA implements the math on torch because no library function covers the RF-specific quantity.

Metric	Task(s)	Definition	Direction	Library
MeanAngularError	`E-LOC-AOA`	Mean wrapped absolute angular error, degrees.	lower	custom on `torch`
BalancedAccuracy	`E-LOC-LOS`	Macro-recall balanced accuracy for class imbalance.	higher	`torchmetrics`
AUC	`E-ID-DRONE`, `E-SC-SENSE`	Area under the ROC (receiver operating characteristic) curve.	higher	`torchmetrics`
Top1Accuracy / EER	`E-ID-FP`	Top-1 closed-set accuracy and equal-error-rate operating point.	higher / lower	`torchmetrics`
PlainAccuracy	`E-ID-AMC`	Multiclass accuracy (continuity column, excluded from OOD-avg).	higher	`torchmetrics`
TopKAccuracy	`E-CH-BEAM`	Top-k categorical accuracy over the beam codebook.	higher	`torchmetrics`
NMSE	`E-CH-CSI`	Normalized mean squared error over channel-tensor energy, dB.	lower	custom on `torch`
SISDR	`E-SC-SEP`	Scale-invariant signal-to-distortion ratio, dB.	higher	`torchmetrics`
AUROC	`E-SC-ANOM`	Area under the ROC curve for anomaly or novelty detection.	higher	`torchmetrics`
Bleu / Meteor / CIDER	`E-S2T-CAP`	N-gram precision, semantic match, and consensus captioning scores.	higher	`torchmetrics` / `nltk`
ExactMatch / F1	`E-S2T-QA`	SQuAD-style (Stanford Question Answering Dataset) exact match and token F1.	higher	`torchmetrics`

Note

PlainAccuracy for E-ID-AMC (automatic modulation classification) is a continuity column: it is reported but excluded from the aggregated OOD-avg (out-of-distribution average) because the task is saturated and single-antenna-solvable. See Task reference and Aggregation.

Mean angular error¶

Computed by MeanAngularError for E-LOC-AOA. AoA (angle of arrival) is physically a continuous quantity, so the error is a regression loss, not a classification accuracy. The angular distance is measured on the circle to handle wraparound: a prediction of 359 degrees against a true 1 degree costs 2 degrees, not 358.

\[ \mathrm{MAE} = \frac{1}{N} \sum_{i=1}^{N} d_{\mathrm{circ}}\!\left(\hat{\theta}_i,\, \theta_i\right), \qquad d_{\mathrm{circ}}\!\left(\hat{\theta}, \theta\right) = \min\!\bigl(\delta,\; 360 - \delta\bigr), \quad \delta = \bigl(|\hat{\theta} - \theta| \bmod 360\bigr) \]

Units. Degrees.
Direction. Lower is better.
Edge cases. Wraparound is handled by the circular distance, not by plain L1 (linear absolute) error. A mean-angle predictor is a documented floor baseline; a real model must beat it one-sided under leave-one-environment-out.
Library. Custom on torch. torchmetrics ships no circular-distance regression metric, so EMMA wraps the wrapped-absolute-difference reduction in the Metric ABC.

Balanced accuracy¶

Computed by BalancedAccuracy for E-LOC-LOS (LoS / NLoS: line-of-sight / non-line-of-sight). Balanced accuracy is macro recall, the mean of per-class recall. It resists the class imbalance that inflates plain accuracy when one path class dominates an environment.

\[ \mathrm{BalAcc} = \frac{1}{C} \sum_{c=1}^{C} \mathrm{recall}_c = \frac{1}{C} \sum_{c=1}^{C} \frac{TP_c}{TP_c + FN_c} \]

Units. Dimensionless, in \([0, 1]\).
Direction. Higher is better.
Edge cases. A majority-class baseline can score high plain accuracy on imbalanced splits; balanced accuracy is the mitigation. The secondary LEAVE_ONE_BAND_OUT (unseen frequency band) axis layers on top of the environment axis.
Library. torchmetrics.classification.MulticlassAccuracy(average="macro").

AUC and AUROC¶

Computed by AUC for E-ID-DRONE (drone detection) and E-SC-SENSE (spectrum sensing), and by AUROC for E-SC-ANOM (anomaly detection). Both report the area under the ROC curve, a threshold-free ranking score for binary detection.

\[ \mathrm{AUC} = \int_{0}^{1} \mathrm{TPR}(t)\, d\mathrm{FPR}(t) = P\!\bigl(\hat{s}_{+} > \hat{s}_{-}\bigr) \]

where \(\mathrm{TPR}\) is the true-positive rate, \(\mathrm{FPR}\) is the false-positive rate, and \(\hat{s}_{+}, \hat{s}_{-}\) are the model’s scores for a positive and a negative sample.

Units. Dimensionless, in \([0, 1]\).
Direction. Higher is better.
Edge cases. AUC is threshold-free, so it is robust to operating-point choice. For E-SC-ANOM, the anomaly class is rare; AUC remains the threshold-free choice over plain accuracy.
Library. torchmetrics.classification.BinaryAUROC; torchmetrics.classification.MulticlassAUROC for the multiclass drone-detection variant.

Top-1 accuracy and equal error rate¶

Computed by Top1Accuracy and EER for E-ID-FP (emitter or device fingerprinting, SEI: specific emitter identification) under leave-one-unit-out. Top-1 is the closed-set recognition accuracy; EER is the threshold-independent operating point at which the false-accept and false-reject rates cross.

\[ \mathrm{Top\text{-}1} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\!\left[\arg\max_{c} \hat{p}_i = y_i\right] \]

\[ \mathrm{EER}: \quad \mathrm{FAR}(\tau^{\*}) = \mathrm{FRR}(\tau^{\*}) \]

where \(\mathrm{FAR}(\tau) = P(\hat{s} \geq \tau \mid \text{impostor})\) is the false-accept rate and \(\mathrm{FRR}(\tau) = P(\hat{s} < \tau \mid \text{genuine})\) is the false-reject rate.

Units. Dimensionless, in \([0, 1]\).
Direction. Top-1 higher is better; EER lower is better.
Edge cases. EER must be interpolated at the crossing, not snapped to the nearest evaluated threshold, to avoid discretization artifacts at small class counts.
Library. torchmetrics.classification.MulticlassAccuracy for top-1; the EER operating point is interpolated from a torchmetrics DET (detection error tradeoff) curve.

Plain accuracy¶

Computed by PlainAccuracy for E-ID-AMC (continuity column) and for the later radar-waveform and micro-Doppler tasks. It is standard multiclass accuracy.

\[ \mathrm{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\!\left[\arg\max_{c} \hat{p}_i = y_i\right] \]

Units. Dimensionless, in \([0, 1]\).
Direction. Higher is better.
Edge cases. Plain accuracy is rejected as a primary metric for the class-imbalanced scored tasks (E-LOC-LOS, E-ID-DRONE); it is used only where the task is balanced by design or where it serves continuity, as for E-ID-AMC.
Library. torchmetrics.classification.MulticlassAccuracy.

Top-k accuracy¶

Computed by TopKAccuracy for E-CH-BEAM (beam management). A prediction counts as correct if the true best beam is among the top-\(k\) predicted beams.

\[ \mathrm{Top\text{-}k} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\!\left[y_i \in \mathrm{top}\text{-}k\!\left(\hat{p}_i\right)\right] \]

Units. Dimensionless, in \([0, 1]\).
Direction. Higher is better.
Edge cases. \(k\) is pinned per release to match the 3GPP (3rd Generation Partnership Project) AI/ML (artificial intelligence or machine learning) beam-reporting codebook size; it is not tuned per submission.
Library. torchmetrics.classification.MulticlassAccuracy(top_k=k).

NMSE¶

Computed by NMSE for E-CH-CSI (CSI: channel state information). The error is normalized by the channel-tensor energy, matching the DeepMIMO convention, and reported in dB.

\[ \mathrm{NMSE} = 10 \log_{10}\!\left( \frac{\sum_{i} \lVert \hat{\mathbf{H}}_i - \mathbf{H}_i \rVert_F^{2}}{\sum_{i} \lVert \mathbf{H}_i \rVert_F^{2}} \right) \quad [\mathrm{dB}] \]

Units. Decibels (dB).
Direction. Lower is better.
Edge cases. The Frobenius-norm normalization is over the channel-tensor energy, not over the batch count, so the score is comparable across studies. A zero-energy reference is guarded against (an all-zero channel is not scored).
Library. Custom on torch. torchmetrics ships no channel-tensor NMSE, so EMMA wraps the ratio-of-Frobenius-norms reduction on torch.

SI-SDR¶

Computed by SISDR for E-SC-SEP (signal separation). SI-SDR (scale-invariant signal-to-distortion ratio) removes the scale ambiguity between estimate and reference before measuring the distortion.

\[ \alpha = \frac{\langle \hat{s},\, s\rangle}{\lVert s\rVert^{2}}, \quad s_{\mathrm{target}} = \alpha\, s, \quad e_{\mathrm{noise}} = \hat{s} - s_{\mathrm{target}} \]

\[ \mathrm{SI\text{-}SDR} = 10 \log_{10}\!\left( \frac{\lVert s_{\mathrm{target}}\rVert^{2}}{\lVert e_{\mathrm{noise}}\rVert^{2}} \right) \quad [\mathrm{dB}] \]

Units. Decibels (dB).
Direction. Higher is better.
Edge cases. A silent or near-zero reference is guarded against, because the projection \(\alpha\) is undefined when \(\lVert s\rVert^{2} = 0\). Such scenes are excluded from the reduction rather than producing an infinite score.
Library. torchmetrics.audio.ScaleInvariantSignalDistortionRatio.

Captioning metrics¶

Computed by Bleu, Meteor, and CIDER for E-S2T-CAP (RF captioning). BLEU (Bilingual Evaluation Understudy) is a precision-based n-gram overlap with a brevity penalty; METEOR (Metric for Evaluation of Translation with Explicit ORdering) adds synonym and stemming matches; CIDEr (Consensus-based Image Description Evaluation) weights n-grams by consensus with the reference set.

\[ \mathrm{BLEU} = \mathrm{BP} \cdot \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right), \qquad \mathrm{BP} = \min\!\bigl(1,\, e^{\,1 - r/c}\bigr) \]

where \(p_n\) is the modified n-gram precision for order \(n\), \(w_n\) its weight, \(c\) the candidate length, and \(r\) the effective reference length.

Units. Dimensionless.
Direction. Higher is better for all three.
Edge cases. Tokenization of RF captions is the open task-side choice; the tokenizer is pinned per release so scores are comparable.
Library. torchmetrics.text.BLEUScore for BLEU; METEOR via nltk.translate.meteor_score; CIDEr via the consensus-weighted reduction.

Exact match and F1¶

Computed by ExactMatch and F1 for E-S2T-QA (RF question answering), following the SQuAD convention. Exact match is 1 only when the normalized prediction equals the reference; F1 is the token-level harmonic mean of precision and recall.

\[ \mathrm{EM} = \mathbf{1}\!\left[\mathrm{norm}(\hat{a}) = \mathrm{norm}(a)\right], \qquad F1 = \frac{2\, P\, R}{P + R} \]

Units. Dimensionless, in \([0, 1]\).
Direction. Higher is better.
Edge cases. Normalization removes articles, punctuation, and case before comparison, matching the SQuAD script. A prediction with no overlapping tokens scores F1 of 0.
Library. torchmetrics.text.ExactMatch for EM; torchmetrics.classification.F1Score on tokenized spans for F1, or the SQuAD-style token F1 reduction.

References¶

Le Roux, Wisdom, Erdogan, and Hershey, “SDR: Half-baked or Well Done?,” ICASSP 2019, arXiv:1811.02508. SI-SDR for E-SC-SEP. (verify)
Papineni et al., “BLEU: a Method for Automatic Evaluation of Machine Translation,” ACL 2002. The BLEU captioning metric.
Banerjee and Lavie, “METEOR: An Automatic Metric for MT Evaluation,” ACL 2005. (verify)
Vedantam et al., “CIDEr: Consensus-based Image Description Evaluation,” CVPR 2015. (verify)
Rajpurkar et al., “SQuAD: 100,000+ Questions for Machine Comprehension of Text,” EMNLP 2016, arXiv:1606.05250. Exact-match and F1 for E-S2T-QA. (verify)
Davis and Goadrich, “The relationship between Precision-Recall and ROC curves,” ICML 2006. AUC and AUROC grounding. (verify)
Brodersen et al., “The balanced accuracy and its posterior distribution,” ICPR 2010. Balanced accuracy for E-LOC-LOS. (verify)
torchmetrics >= 0.11; nltk (METEOR reduction); torch >= 2.0. Library function paths cited per subsection.