OOD protocol

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The OOD (out-of-distribution) protocol is the evaluation mechanic that makes the score a transfer score rather than in-distribution accuracy. Each task holds out one controlled dimension at evaluation time, scores the model on the held-out value, and reports the mean over held-out values. This page is the normative contract for the OODProtocol ABC and its three concrete protocols. The narrative motivation lives in Generalization; the per-task axis wiring lives in Task reference and Environment reference.

Leave-one-X-out

Let \(X\) be the OOD axis (environment, device, or frequency band). The training split spans a set of values \(\mathcal{X}_{\text{train}}\) on \(X\). Evaluation holds out one value \(x^{\*} \notin \mathcal{X}_{\text{train}}\) at a time, scores the model on every scene with axis value \(x^{\*}\), and averages over the held-out set \(\mathcal{X}_{\text{eval}}\):

\[ \text{score} = \frac{1}{|\mathcal{X}_{\text{eval}}|} \sum_{x \in \mathcal{X}_{\text{eval}}} m\!\bigl(\text{model},\, x\bigr) \]

where \(m(\text{model}, x)\) is the per-task metric (see Metric reference) computed on the scenes whose axis value is \(x\). A model that overfits the statistics of the training values cannot hide: every contribution to the mean comes from a value it never trained on.

The protocols

The OODProtocol ABC defines the fold-construction interface. Three concrete protocols implement it, one per axis EMMA v0.1 uses.

Protocol

Axis

Holds out

Used by

LeaveOneEnvironmentOut

ENVIRONMENT

One channel environment.

E-LOC-AOA, E-LOC-LOS, E-ID-DRONE

LeaveOneUnitOut

DEVICE

Every capture of one device.

E-ID-FP

LeaveOneBandOut

FREQUENCY_BAND

One frequency band.

E-LOC-LOS (secondary axis)

LeaveOneEnvironmentOut answers “does the model transfer across channel conditions?”. LeaveOneUnitOut answers “does the model recognize a device it has never seen, without confounding the capture session?”. LeaveOneBandOut layers an unseen-band stress on top. The three are not interchangeable; each task pins the axis that matches what it measures, recorded in Task reference.

Leave-one-environment-out

Training spans \(\mathcal{E}_{\text{train}}\) (for v0.1, at least E_URBAN and E_RURAL). The reported score is:

\[ \text{score} = \frac{1}{|\mathcal{E}_{\text{eval}}|} \sum_{e \in \mathcal{E}_{\text{eval}}} m(\text{model}, e) \]

Leave-one-unit-out

Training spans a set of device units \(\mathcal{U}_{\text{train}}\). Every capture of the held-out unit \(u^{\*}\) goes to the eval fold, so a model cannot have seen any capture session of \(u^{\*}\) at train time. This isolates device identity from capture-session and receiver-front-end confounds, conditional on the recipe randomizing the receiver chain across units (the open recipe question recorded in Tasks validation).

Leave-one-band-out

Training spans a set of frequency bands \(\mathcal{B}_{\text{train}}\) (for example 2.4 GHz). The held-out band \(b^{\*}\) (for example 5.8 GHz) is unseen at train time. This layers onto the environment axis for E-LOC-LOS.

Protocol runner pseudo-code

The ProtocolRunner materializes one fold per held-out value, fits the readout head on the training fold, scores the eval fold, and averages. The frozen backbone is shared across folds; only the lightweight readout head is re-fit.

def run_leave_one_out(backbone, protocol, scenes, metric):
    """Leave-one-X-out over the protocol axis. Returns the mean score."""
    scores = []
    for x_hold in protocol.eval_values():
        train = scenes.filter(protocol.axis != x_hold)
        eval  = scenes.filter(protocol.axis == x_hold)
        head = protocol.readout_head().fit(backbone, train)
        scores.append(metric(head.predict(eval), eval.labels))
    return mean(scores)

  • The frozen backbone (weights not updated) is constant across folds.

  • The readout head is re-fit per fold, because the training scenes differ per holdout.

  • The metric reduction is the mean over held-out values, matching the formula above.

Open methodological question: low power at two environments

(Severity: high, open, deferred to open questions) v0.1 ships with at least two synthetic environments (E_URBAN, E_RURAL), so leave-one-environment-out has \(N = 2\) for the environment-level statistic. A two-point mean has essentially no power to distinguish models at the environment stratum, and a single outlier environment dominates the aggregate. The proposed mitigations are: add a third environment as soon as the recipe allows; report per-environment scores alongside every aggregate, never the aggregate alone; and treat the v0.1 aggregate as a comparative ranking under a fixed protocol, not a population estimate of transfer. This is the sharpest methodology threat to the v0.1 leaderboard and is named explicitly rather than hidden. See Tasks validation and Harness validation.

References

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. The leave-one-X-out framing and generalization-first precedent.

  • Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” NeurIPS 2019, arXiv:1906.02530. Predictive performance under controlled shift. (verify)

See Also