Leaderboard row¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

A leaderboard row is the public result a scored submission produces. It carries the per-task metric columns, the headline OOD-avg (out-of-distribution average), the sim-to-real gap, and the audit hashes that make the result an auditable triple of (model, data, metric). The serialized form is the LeaderboardRow Pydantic model, posted by HoldoutScorer via ScoringServer.

The row is not a self-reported number. The data half of the triple is the data_release_hash recorded directly on the row; the metric half is the published scoring code and its frozen normalization anchors (see Aggregation); the model half is the set of prediction-bundle hashes that pin exactly which outputs were scored. A reviewer can re-derive the data from the recipe, re-score the predictions, and check the number.

Schema fields¶

Field	Type	Purpose
`rank`	`int`	Position on the public board, sorted by `ood_avg` (see Sort order).
`model`	`str`	Model display name, copied from the submission’s `model_name`.
`submitter`	`str`	Submitter handle or team name.
`task_scores`	`dict[str, float]`	Per-task metric values, keyed by TaskID value (see Per-task columns).
`ood_avg`	`float`	Normalized OOD average across the scored tasks; the headline number.
`sim_real_gap`	`float`	Sim-to-real gap in percentage points; the fidelity column.
`data_release_hash`	`str`	Content hash of the DataReleaseManifest the scores were computed on.

For audit, the public board also surfaces the prediction-bundle hashes of the scored submission, so the model half of the triple is visible alongside the data half. Those hashes are carried on the linked RunManifest (one per task), not on the row itself; a reviewer recovers them from the run record.

Per-task columns¶

The task_scores dict renders as one column per task on the public board. Metric acronyms: MAE is mean angular error, AUC is area under the ROC (receiver operating characteristic) curve, AMC is automatic modulation classification, Acc is accuracy, pp is percentage points.

Column	Task	Metric	Direction
`E-LOC-AOA`	Angle-of-arrival / DoA (direction-of-arrival)	MAE (degrees)	lower is better
`E-LOC-LOS`	LoS / NLoS (line-of-sight / non-line-of-sight)	Acc (balanced accuracy)	higher is better
`E-ID-DRONE`	Multi-antenna RF (radio-frequency) drone detection	AUC	higher is better
`E-ID-FP`	Emitter / device fingerprinting (SEI: specific emitter identification)	Top-1 accuracy	higher is better
`E-ID-AMC`†	AMC (continuity)	Acc (plain accuracy)	higher is better

† E-ID-AMC is a continuity column, reported but excluded from ood_avg. It is saturated, single-antenna-solvable, and its publisher states the data has known errata and is not used in its products, so including it would dilute the transfer signal the aggregate measures. See Aggregation.

The sim-to-real gap is reported in its own sim_real_gap column, never folded into ood_avg. The two numbers answer different questions: OOD-avg measures controlled transfer; the gap measures fidelity to real captures. A model can rank first on OOD-avg and still carry a large gap. See Sim-to-real gap.

Sort order¶

Rows are sorted by ood_avg, descending (higher is better). Ties are broken by the smaller sim_real_gap (a smaller gap is better). A model that tops the synthetic columns but carries a large sim-to-real gap is therefore ranked below a tied model whose competence transfers to real captures.

Example¶

{
  "rank": 1,
  "model": "my-fm@v1",
  "submitter": "team-alpha",
  "task_scores": {
    "E_LOC_AOA": 5.1,
    "E_LOC_LOS": 0.94,
    "E_ID_DRONE": 0.95,
    "E_ID_FP": 0.82,
    "E_ID_AMC": 0.60
  },
  "ood_avg": 0.83,
  "sim_real_gap": 3.0,
  "data_release_hash": "sha256:c1d4...9a02"
}

The illustrative numbers above match the placeholder board on the Leaderboard concept page; they are not real results.

References¶

Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. The single aggregated score the ood_avg column generalizes. (verify)
Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Results-only evaluation under controlled distribution shift; the sort-by-transfer precedent.
Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Auditable result records at benchmark scale; the audit-hash contract the row carries.