Metrics¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

Per-task metrics, aggregators, and the sim-to-real gap. Every metric reuses torchmetrics, scipy, or numpy wherever a library already defines it. A metric is flagged custom only when no library implementation covers the RF (radio-frequency)-specific quantity; the three custom cases (MeanAngularError, NMSE, and SimToRealGap) wrap a minimal computation on torch tensors. The per-metric formulas, edge cases, and library paths live in Metric reference; this page is the per-class Python contract.

Three roles¶

Role	Examples
ABC	Metric, Aggregator
Concrete metrics	MeanAngularError, BalancedAccuracy, and the rest of the per-task metric classes
Concrete aggregators	OODAvg
Fidelity-gap reporter	SimToRealGap, a standalone class (not an Aggregator; its pairwise `(synth_score, real_score)` signature is incompatible with the `__call__(fold_scores)` fold-reduction contract)

Class index¶

Class	Role	Direction	Wraps
Metric	ABC	varies	Defines the `compute` contract
MeanAngularError	Metric	lower	custom on `torch`
BalancedAccuracy	Metric	higher	`torchmetrics`
AUC	Metric	higher	`torchmetrics`
Top1Accuracy	Metric	higher	`torchmetrics`
EER	Metric	lower	custom on `torch` from ROC
PlainAccuracy	Metric	higher	`torchmetrics`
TopKAccuracy	Metric	higher	`torchmetrics`
NMSE	Metric	lower	custom on `torch`
SISDR	Metric	higher	`torchmetrics`
AUROC	Metric	higher	`torchmetrics`
Bleu	Metric	higher	`torchmetrics`
Meteor	Metric	higher	`nltk`
CIDER	Metric	higher	consensus reduction
ExactMatch	Metric	higher	`torchmetrics`
F1	Metric	higher	`torchmetrics`
Aggregator	ABC	varies	Defines the fold-aggregation contract
OODAvg	Aggregator	higher	Normalizes and averages per-task scores
SimToRealGap	Reporter	lower	Reports synthetic minus real-capture score

`class emma.metrics.Metric(ABC)`¶

The per-task metric contract. Each concrete metric wraps a standard library function and exposes a uniform compute interface so the harness can swap metrics without touching call sites.

class Metric(ABC):
    direction: ClassVar[MetricDirection]

    @abstractmethod
    def compute(
        self,
        predictions: torch.Tensor,
        references: torch.Tensor,
        split: Split,
    ) -> float: ...

Class attributes¶

Attribute	Type	Purpose
`direction`	MetricDirection	Optimization sense; consumed by OODAvg to decide inversion

Abstract methods¶

`compute(predictions, references, split)`¶

Parameter	Type	Purpose
`predictions`	`torch.Tensor`	Model outputs for the fold
`references`	`torch.Tensor`	Ground-truth labels for the fold
`split`	Split	The split being scored

Returns. float, the raw metric value in the unit documented for the task.

Notes¶

Subclasses set direction as a ClassVar. A lower-is-better metric sets direction = MetricDirection.LOWER.
The harness calls compute once per held-out fold; the Aggregator reduces the fold scores.

Concrete metrics¶

`class emma.metrics.MeanAngularError(Metric)`¶

Mean wrapped absolute angular error for E_LOC_AOA, in degrees. Custom on torch because torchmetrics ships no circular-distance regression metric. See Metric reference for the formula.

class MeanAngularError(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.LOWER

    def compute(self, predictions, references, split) -> float: ...

Wraps the circular distance min(delta, 360 - delta) so a prediction of 359 degrees against a true 1 degree costs 2 degrees, not 358.

`class emma.metrics.BalancedAccuracy(Metric)`¶

Macro-recall balanced accuracy for E_LOC_LOS. Resists the class imbalance that inflates plain accuracy when one path class dominates.

class BalancedAccuracy(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps torchmetrics.classification.MulticlassAccuracy(average="macro").

`class emma.metrics.AUC(Metric)`¶

Area under the ROC (receiver operating characteristic) curve for E_ID_DRONE and E_SC_SENSE. Threshold-free ranking score that supports both the binary spectrum-sensing case and the multiclass drone-detection case.

class AUC(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def __init__(self, num_classes: int) -> None: ...
    def compute(self, predictions, references, split) -> float: ...

Parameter	Type	Default	Purpose
`num_classes`	`int`	required	Number of target classes; `2` selects `torchmetrics.classification.BinaryAUROC`, greater than 2 selects `torchmetrics.classification.MulticlassAUROC`

Wraps torchmetrics.classification.BinaryAUROC for the binary variant and MulticlassAUROC for the drone-detection multiclass variant. The underlying torchmetrics metric is fixed at construction time from num_classes.

`class emma.metrics.Top1Accuracy(Metric)`¶

Top-1 closed-set accuracy for E_ID_FP.

class Top1Accuracy(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps torchmetrics.classification.MulticlassAccuracy.

`class emma.metrics.EER(Metric)`¶

Equal error rate for E_ID_FP. The threshold-independent operating point at which false-accept and false-reject rates cross.

class EER(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.LOWER

    def compute(self, predictions, references, split) -> float: ...

Custom on torch: built from the binary ROC curve returned by torchmetrics.classification.BinaryROC by interpolating to the threshold where the false-accept rate equals the false-reject rate. torchmetrics ships no direct EER primitive at the time of writing (verify).

`class emma.metrics.PlainAccuracy(Metric)`¶

Multiclass accuracy for E_ID_AMC (continuity column) and later radar-waveform tasks.

class PlainAccuracy(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps torchmetrics.classification.MulticlassAccuracy. Excluded from the aggregated OOD average for E_ID_AMC.

`class emma.metrics.TopKAccuracy(Metric)`¶

Top-k categorical accuracy over the beam codebook for E_CH_BEAM.

class TopKAccuracy(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def __init__(self, k: int = 5) -> None: ...
    def compute(self, predictions, references, split) -> float: ...

Parameter	Type	Default	Purpose
`k`	`int`	`5`	Number of top beams counted as correct; pinned per release to match the 3GPP (3rd Generation Partnership Project) AI/ML (artificial intelligence or machine learning) codebook

Wraps torchmetrics.classification.MulticlassAccuracy(top_k=k).

`class emma.metrics.NMSE(Metric)`¶

Normalized mean squared error for E_CH_CSI, in decibels. Custom on torch because torchmetrics ships no channel-tensor NMSE.

class NMSE(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.LOWER

    def compute(self, predictions, references, split) -> float: ...

Wraps the ratio-of-Frobenius-norms reduction matching the DeepMIMO convention. Expressed in decibels as 10 * log10(||prediction||_F^2 / ||reference||_F^2); see Metric reference for the per-element reduction.

`class emma.metrics.SISDR(Metric)`¶

Scale-invariant signal-to-distortion ratio for E_SC_SEP, in decibels.

class SISDR(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps torchmetrics.audio.ScaleInvariantSignalDistortionRatio.

`class emma.metrics.AUROC(Metric)`¶

Area under the ROC curve for E_SC_ANOM anomaly detection.

class AUROC(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps torchmetrics.classification.BinaryAUROC. Distinct from AUC only in scope: AUROC is strictly binary (anomaly versus nominal), while AUC also covers the multiclass drone-detection variant. Both reduce to AUROC math in the binary case.

`class emma.metrics.Bleu(Metric)`¶

BLEU (Bilingual Evaluation Understudy) n-gram precision for E_S2T_CAP captioning.

class Bleu(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def __init__(self, n: int = 4) -> None: ...
    def compute(self, predictions, references, split) -> float: ...

Parameter	Type	Default	Purpose
`n`	`int`	`4`	Maximum n-gram order

Wraps torchmetrics.text.BLEUScore.

`class emma.metrics.Meteor(Metric)`¶

METEOR (Metric for Evaluation of Translation with Explicit ORdering) semantic-match score for E_S2T_CAP.

class Meteor(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps nltk.translate.meteor_score.

`class emma.metrics.CIDER(Metric)`¶

CIDEr (Consensus-based Image Description Evaluation) captioning score for E_S2T_CAP.

class CIDER(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps the consensus-weighted n-gram reduction. EMMA targets the canonical CIDEr-D variant from Vedantam et al.; the implementation is pycocoevalcap (verify) until a torchmetrics equivalent ships.

`class emma.metrics.ExactMatch(Metric)`¶

Exact match for E_S2T_QA question answering, following the SQuAD (Stanford Question Answering Dataset) scoring convention.

class ExactMatch(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps torchmetrics.text.ExactMatch.

`class emma.metrics.F1(Metric)`¶

Token-level F1 for E_S2T_QA, following the SQuAD convention.

class F1(Metric):
    direction: ClassVar[MetricDirection] = MetricDirection.HIGHER

    def compute(self, predictions, references, split) -> float: ...

Wraps torchmetrics.classification.F1Score on tokenized spans.

Aggregators¶

`class emma.metrics.Aggregator(ABC)`¶

The fold-aggregation contract. Reduces per-fold metric values into a single reported number.

class Aggregator(ABC):
    @abstractmethod
    def __call__(self, fold_scores: dict[str, float]) -> float: ...

Abstract methods¶

Method	Returns	Purpose
`__call__(fold_scores)`	`float`	Reduce per-fold scores to a single value

`class emma.metrics.OODAvg(Aggregator)`¶

The headline leaderboard number: the OOD (out-of-distribution) average. Normalizes each task score to a 0 to 1 range with a frozen per-release anchor pair, inverts lower-is-better metrics, and averages over the aggregated task set. See Aggregation for the formula.

class OODAvg(Aggregator):
    def __init__(self, anchors: dict[str, tuple[float, float]]) -> None: ...

    def __call__(self, fold_scores: dict[str, float]) -> float: ...

Constructor parameters¶

Parameter	Type	Purpose
`anchors`	`dict[str, tuple[float, float]]`	Frozen per-task anchor pair `(m_min, m_max)`. Never recomputed per submission

Invariants¶

Anchors are frozen per data release and published with the release manifest. A submission cannot move the normalization.
Raw scores outside the anchor range are clipped to 0 or 1 after normalization.
E_ID_AMC is excluded from the aggregated task set.

`class emma.metrics.SimToRealGap`¶

The fidelity column. Reports the difference between a model’s score on the synthetic split and its score on the real-capture OOD subset, in percentage points.

class SimToRealGap:
    def __init__(self, anchors: dict[str, tuple[float, float]]) -> None: ...

    def __call__(self, synth_score: float, real_score: float) -> float: ...

SimToRealGap is a standalone fidelity-gap reporter. It does not inherit Aggregator because its pairwise (synth_score, real_score) call signature is not substitutable for the __call__(fold_scores: dict[str, float]) -> float fold-reduction contract; subclassing Aggregator would be a Liskov violation. OODAvg is the sole Aggregator subclass.

Constructor parameters¶

Parameter	Type	Purpose
`anchors`	`dict[str, tuple[float, float]]`	Frozen per-task anchor pair, shared with OODAvg

Notes¶

Returns 100 * (synth_normalized - real_normalized), in percentage points (pp).
A positive gap means the model scored better on synthetic rfgen scenes than on real captures. See Sim-to-real gap.
The gap is reported alongside OOD average, never folded into it.

References¶

Library: torchmetrics >= 1.0 (torchmetrics.classification.MulticlassAccuracy, torchmetrics.classification.BinaryAUROC, torchmetrics.classification.MulticlassAUROC, torchmetrics.classification.BinaryROC, torchmetrics.classification.F1Score, torchmetrics.audio.ScaleInvariantSignalDistortionRatio, torchmetrics.text.BLEUScore, torchmetrics.text.ExactMatch). (verify)
Library: nltk >= 3.8 (nltk.translate.meteor_score). (verify)
Library: pycocoevalcap (CIDEr-D) for CIDER. (verify)
Library: torch >= 2.1 (custom reductions for MeanAngularError, NMSE, and EER).
Convention: DeepMIMO NMSE convention (ratio of Frobenius norms); see Metric reference for the source citation.
Convention: SQuAD (Stanford Question Answering Dataset) exact-match and token-F1 scoring for ExactMatch and F1.

Metrics¶

Three roles¶

Class index¶

class emma.metrics.Metric(ABC)¶

Class attributes¶

Abstract methods¶

compute(predictions, references, split)¶

Notes¶

Concrete metrics¶

class emma.metrics.MeanAngularError(Metric)¶

class emma.metrics.BalancedAccuracy(Metric)¶

class emma.metrics.AUC(Metric)¶

class emma.metrics.Top1Accuracy(Metric)¶

class emma.metrics.EER(Metric)¶

class emma.metrics.PlainAccuracy(Metric)¶

class emma.metrics.TopKAccuracy(Metric)¶

class emma.metrics.NMSE(Metric)¶

class emma.metrics.SISDR(Metric)¶

class emma.metrics.AUROC(Metric)¶

class emma.metrics.Bleu(Metric)¶

class emma.metrics.Meteor(Metric)¶

class emma.metrics.CIDER(Metric)¶

class emma.metrics.ExactMatch(Metric)¶

class emma.metrics.F1(Metric)¶

Aggregators¶

class emma.metrics.Aggregator(ABC)¶

Abstract methods¶

class emma.metrics.OODAvg(Aggregator)¶

Constructor parameters¶

Invariants¶

class emma.metrics.SimToRealGap¶

Constructor parameters¶

Notes¶

References¶

See Also¶

`class emma.metrics.Metric(ABC)`¶

`compute(predictions, references, split)`¶

`class emma.metrics.MeanAngularError(Metric)`¶

`class emma.metrics.BalancedAccuracy(Metric)`¶

`class emma.metrics.AUC(Metric)`¶

`class emma.metrics.Top1Accuracy(Metric)`¶

`class emma.metrics.EER(Metric)`¶

`class emma.metrics.PlainAccuracy(Metric)`¶

`class emma.metrics.TopKAccuracy(Metric)`¶

`class emma.metrics.NMSE(Metric)`¶

`class emma.metrics.SISDR(Metric)`¶

`class emma.metrics.AUROC(Metric)`¶

`class emma.metrics.Bleu(Metric)`¶

`class emma.metrics.Meteor(Metric)`¶

`class emma.metrics.CIDER(Metric)`¶

`class emma.metrics.ExactMatch(Metric)`¶

`class emma.metrics.F1(Metric)`¶

`class emma.metrics.Aggregator(ABC)`¶

`class emma.metrics.OODAvg(Aggregator)`¶

`class emma.metrics.SimToRealGap`¶