Aggregation

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The OOD-avg (out-of-distribution average) is the headline leaderboard number: a single normalized score that ranks frozen backbones across the v0.1 task set. It is computed by OODAvg, the Aggregator implementation. Each task score is first reduced to a \([0, 1]\) range by a frozen normalization, lower-is-better metrics are inverted, and the results are averaged. The narrative motivation lives in Generalization; the per-task metrics live in Metric reference.

The aggregated task set

The v0.1 aggregate spans the four scored tasks:

\[ \mathcal{T} = \{\texttt{E-LOC-AOA},\; \texttt{E-LOC-LOS},\; \texttt{E-ID-DRONE},\; \texttt{E-ID-FP}\} \]

E-ID-AMC (automatic modulation classification) is reported but excluded. It is a continuity column: saturated, single-antenna-solvable, and its publisher states the data has known errata and is not used in its products, so including it would dilute the transfer signal the aggregate is meant to surface. See Task reference.

Normalization

Each raw task score \(m\) is mapped to \(\hat{m} \in [0, 1]\) with a frozen per-release anchor pair \((m_{\min}, m_{\max})\). Lower-is-better metrics are inverted first so that higher is uniformly better:

\[\begin{split} \hat{m} = \begin{cases} \dfrac{m - m_{\min}}{m_{\max} - m_{\min}} & \text{higher is better} \\[6pt] \dfrac{m_{\max} - m}{m_{\max} - m_{\min}} & \text{lower is better} \end{cases} \end{split}\]

The OOD-avg is the mean over the aggregated task set:

\[ \text{OOD-avg} = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \hat{m}_t \]

  • Units. Dimensionless, in \([0, 1]\). A perfect model across all four tasks scores 1.

  • Direction. Higher is better.

  • Per-task directions. E-LOC-AOA is lower-is-better (angular error); the other three are higher-is-better. The inversion branch handles the difference.

Frozen anchors

The anchor pair \((m_{\min}, m_{\max})\) is frozen per task at each data release and recorded in the release manifest. Anchors are never recomputed per submission. This is an anti-gaming property: a submission cannot move the normalization to flatter its own scores, because the anchors are fixed before any submission is scored and are published with the release.

  • \(m_{\min}\) is the lower bound (chance or floor baseline) for the task.

  • \(m_{\max}\) is the upper bound (a strong reference or the theoretical ceiling) for the task.

  • A raw score outside \([m_{\min}, m_{\max}]\) is clipped to \([0, 1]\) after normalization, so an out-of-range result saturates rather than shifting the anchors.

The exact anchor values finalize and freeze with the first v0.1 data release; until then they are proposed-contract.

What OOD-avg is not

OOD-avg aggregates the synthetic OOD scores only. The sim-to-real gap, computed by SimToRealGap, is reported as a separate column and is never folded into the aggregate. See Sim-to-real gap. A model can rank first on OOD-avg and still carry a large gap; the two numbers answer different questions.

References

  • Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. One frozen backbone, lightweight readouts, one aggregated score; the structural precedent for OOD-avg. (verify)

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Generalization-first aggregation; the leave-one-X-out scores that feed the aggregate.

See Also