Leaderboard validation

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

Scope: the six-lens validation of the leaderboard layer, concentrated on the ResultsOnlyBoard store, the HoldoutScorer scoring path, the SubmissionValidator and AntiOverfit guards, and the sim-to-real gap threshold. The anti-gaming algorithm is audited in Anti-overfit; this report asks whether the reporting surface as a whole produces trustworthy rankings.

1. Purpose and construct

EMMA (Electromagnetic Multi-task Model Assessment) scores predictions against a secret holdout via HoldoutScorer, posts the result as a LeaderboardRow, and protects both the holdout and the ranking with the anti-overfit and validation guards. The construct claim is that a posted rank is a trustworthy measure of a model’s transfer competence, not a measure of who gamed hardest or who graded their own model. Construct validity asks four questions: does the held-out resist overfitting; is the sim-to-real gap threshold principled; is the operator’s own model re-scored by a third party; and is the real-capture subset large enough for a stable gap.

  • Claim. A leaderboard row reflects model competence on an unseen held-out, scored by code the submitter cannot influence, with the operator’s own model held to the same standard as an external submitter’s.

  • Proxy. ScoringServer validates a Submission, checks it with AntiOverfit, scores it with HoldoutScorer, and posts a LeaderboardRow carrying the data_release_hash alongside a linked RunManifest whose per-task prediction_bundle_hashes carry the model half of the audit triple.

  • Omissions. The v0.1 board is results-only; it does not verify the submitter’s training compute or data. Full sandboxed-code submission arrives at v1.

  • Downstream risk. A board whose holdout can be probed, whose threshold is tuned per submission, or whose operator self-grades loses the neutrality property that is the benchmark’s reason to exist.

  • Domain utility. The board is the artifact downstream communities (positioning, defense, spectrum, communications) cite as the result. Its trustworthiness is the load-bearing property for adoption.

Sharpest construct threats

(Severity: high, architectural) Neutrality cannot be manufactured retroactively. Once a model-builder has graded its own model on its own private test set, no later ring-fence undoes the conflict. The operator’s model must therefore be re-scored by an external co-steward on every release, recorded in a ReScoringLog, from day one. This is the single property no model vendor can replicate, and it is the construct threat the governance design exists to close.

(Severity: medium) The sim-to-real gap threshold is proposed-contract. A threshold tuned per submission re-imports self-grading through the back door; a threshold frozen before the first real-capture scores is principled but uncalibrated. The construct depends on the threshold being chosen by a fixed rule and frozen at release.

2. Mathematical fidelity

The leaderboard layer records formula outputs; it does not recompute them. The per-task metrics and the OOD-avg (out-of-distribution average) normalization are audited in Metrics validation and Aggregation, where OODAvg is the sole Aggregator. SimToRealGap is a standalone fidelity-gap reporter, not an Aggregator: it emits a parallel column the board surfaces alongside the aggregate, never folded into it.

  • Audit triple. A posted result is an auditable triple of (model, data, metric): the model half is the per-task prediction_bundle_hashes on the linked RunManifest, the data half is the data_release_hash on the LeaderboardRow, and the metric half is the row’s task_scores plus the published scoring code and its frozen normalization anchors. (Severity: low) The row itself does not carry the prediction-bundle hashes; a reviewer recovers them from the run record. See Leaderboard row and Run manifest.

  • Gap formula. SimToRealGap computes \(\Delta_{\text{s2r}} = 100 \cdot (\hat{m}_{\text{synth}} - \hat{m}_{\text{real}})\), in percentage points, with both scores mapped to \([0, 1]\) by the shared OOD-avg normalization. (Severity: low) Covered in Sim-to-real gap; the leaderboard must record the gap as reported by the metric, not recompute it from rounded per-task scores.

  • Rank computation. Rank is by ood_avg descending, ties broken by the smaller sim_real_gap. A tie on both must be resolved deterministically (for example by the earlier submission timestamp), not by insertion order. (Severity: low) Open: pin the deterministic tiebreak beyond the gap.

3. Empirical realism

The realism question for the leaderboard is whether the real-capture OOD (out-of-distribution) subset (EMMA-REAL-OOD) is large enough to produce a stable sim-to-real estimate. The subset is on the order of 8,000 captures, drawn from the three public testbeds (Colosseum, POWDER, COSMOS), and exists to measure fidelity, not to train on.

  • Coverage. With roughly 8,000 captures split across three testbeds and several tasks, a per-task real score is computed on the order of hundreds to low thousands of scenes. The standard error of that score propagates into the gap: \(\mathrm{Var}(\Delta_{\text{s2r}}) = 10^4 \cdot (\mathrm{Var}(\hat{m}_{\text{synth}}) + \mathrm{Var}(\hat{m}_{\text{real}}))\) under independence. If the resulting confidence interval (CI) on the gap is wider than the gap threshold, the fidelity flag is noisier than the decision it drives.

  • Stability criterion. The subset size is frozen at release only when the bootstrap CI on the gap is narrower than the proposed threshold. If 8,000 captures do not meet that bar for every scored task, either the threshold must widen or the subset must grow before the gap column is reported as a decision rather than an estimate.

(Severity: medium, open) Whether 8,000 captures is enough is an empirical question that cannot be answered until the real-capture scores exist. The construct-preserving posture is to report the gap with its CI, freeze the subset size when the CI tightens, and never report a gap flag without the CI that justifies it. Cross-link: Datasets validation and Sim-to-real gap.

4. Literature grounding

The canonical sources, drawn from the master reading list:

  • Neutrality and re-scoring. Mattson et al. (MLPerf) and Russakovsky et al. (ImageNet) are the seeded-with-baselines, governance-follows-adoption precedents; the ReScoringLog is the ring-fence instrument.

  • Pre-registration. The ICMJE clinical-trial registration analogy grounds the operator seed-backbone declaration before holdout unseal.

  • Anti-gaming. Tramer et al. (stealing models via prediction APIs) grounds the residual adversarial-query-extraction risk the AntiOverfit query auditor monitors.

  • Sim-to-real. “RF-Analyzer” concedes sim-to-real generalization breaks in low-SNR (signal-to-noise ratio) and OOD regimes; the load-bearing caveat the gap column surfaces.

(Severity: low) The Tramer and ICMJE entries carry (verify) in the reading list; none of their claims are presented as fact here until verified. No further literature entries are required for this report.

5. Experimental methodology and planned tests

Tests are written so an implementer can port them directly to tests/validation/leaderboard/ once src/emma/ lands.

Duplicate-row rejection - tests/validation/leaderboard/test_duplicate_rejected.py

  • Falsifiable claim. A submission whose every per-task prediction-bundle hash matches the linked RunManifest of an existing posted row is rejected, not re-scored.

  • Failure mode. A submitter re-posts a prior row to farm rank, or two teams post identical predictions.

  • Design. Post a row; submit again with the same prediction_bundle_hashes recorded on the run manifest; assert ScoringServer.submit raises SubmissionRejected with a duplicate reason.

  • Sample size. One duplicated submission per task set.

  • Statistical test. Assertion-based; a duplicate must never produce a second row.

  • Tolerance. Zero tolerance.

  • Gold-standard reference. The per-task prediction_bundle_hashes on the existing row’s linked run manifest.

Schema-rejection test - tests/validation/leaderboard/test_schema_rejected.py

  • Falsifiable claim. A prediction bundle with the wrong schema is rejected before scoring.

  • Failure mode. A malformed bundle (missing scene_id, wrong columns, probability vector not summing to one) is scored anyway, producing a garbage row.

  • Design. Submit a bundle whose Parquet schema violates the task contract (see Prediction bundle); assert SubmissionValidator.validate raises PredictionSchemaError and no row is posted.

  • Sample size. One malformed bundle per violation class.

  • Statistical test. Assertion-based.

  • Tolerance. Zero tolerance; a malformed bundle must never be scored.

  • Gold-standard reference. The per-task prediction contract.

Re-scoring equivalence test - tests/validation/leaderboard/test_rescoring_matches.py

  • Falsifiable claim. An external co-steward re-scoring the same submission produces identical numbers to the operator’s run.

  • Failure mode. The operator’s posted score and the co-steward’s re-score disagree beyond floating-point tolerance, indicating non-determinism or a hidden configuration drift.

  • Design. Run HoldoutScorer on a submission; independently re-run on the same bundle and release; record both in ReScoringLog entries; assert the scores match.

  • Sample size. Every operator submission, every release.

  • Statistical test. Exact equality within relative tolerance 1e-6 on each task metric.

  • Tolerance. Relative tolerance 1e-6.

  • Gold-standard reference. The operator’s own run manifest.

Rate-limit and audit tests - tests/validation/leaderboard/test_rate_limit.py, test_query_audit.py

  • Falsifiable claim. A submitter exceeding the rate cap is rejected, and a near-duplicate probe sequence is flagged.

  • Failure mode. Brute-force re-submission or query probing goes undetected.

  • Design. Submit beyond max_submissions_per_day; assert AntiOverfit.check raises SubmissionRejected with a rate_limit reason. Then submit a sequence of near-duplicate bundles; assert a context reason fires.

  • Sample size. A bounded probe sequence.

  • Statistical test. Assertion-based.

  • Tolerance. The rate cap is exact; the query-audit threshold is pinned per release.

  • Gold-standard reference. The AntiOverfit defaults.

Gap-stability probe - tests/validation/leaderboard/test_gap_stability.py

  • Falsifiable claim. The sim-to-real gap CI from the real-capture subset is narrower than the proposed threshold.

  • Failure mode. The CI is wider than the threshold, so the gap flag is noisier than the decision.

  • Design. Bootstrap the per-scene real-capture scores to a 95% CI on \(\hat{m}_{\text{real}}\), propagate to a CI on the gap, compare to the threshold.

  • Sample size. The EMMA-REAL-OOD subset (on the order of 8,000 captures).

  • Statistical test. Bootstrap 95% CI on the gap; one-sided that the CI half-width is below the threshold.

  • Tolerance. CI half-width below the frozen threshold.

  • Gold-standard reference. The frozen SimToRealGap threshold recorded at release.

Rank-ordering test - tests/validation/leaderboard/test_rank_ordering.py

  • Falsifiable claim. Rows are sorted by ood_avg descending, ties broken by the smaller sim_real_gap, then by the earlier submission timestamp.

  • Failure mode. Two rows with near-identical ood_avg and sim_real_gap are ordered by insertion order, leaking nondeterminism into the public board.

  • Design. Construct three rows covering each tiebreak layer (clear win on ood_avg; tie on ood_avg broken by gap; tie on both broken by timestamp); assert the posted order matches the rule.

  • Sample size. Three constructed rows.

  • Statistical test. Assertion-based.

  • Tolerance. Zero tolerance on ordering.

  • Gold-standard reference. The sort rule in Leaderboard row.

Frozen-anchor read test - tests/validation/leaderboard/test_frozen_anchor_read.py

  • Falsifiable claim. The OOD-avg normalization anchors and the sim-to-real gap threshold the scorer loads at scoring time are byte-for-byte equal to the frozen release artifacts.

  • Failure mode. A stale or patched anchor file silently shifts scores or the gap decision between submissions on the same release.

  • Design. Load the anchors and threshold the scorer reads; hash-compare to the pinned release artifacts; assert equality and that the row’s ood_avg and sim_real_gap derive from those exact values.

  • Sample size. One read per release.

  • Statistical test. Assertion-based; hash equality on the anchor and threshold artifacts.

  • Tolerance. Zero tolerance.

  • Gold-standard reference. The pinned release anchor and threshold artifacts.

6. Robustness boundaries

  • Adversarial query extraction. (Severity: medium, monitored) A submitter with enough submissions can in principle extract partial held-out information from score feedback (Tramer et al.). The rate limit bounds the query budget and the query auditor flags the attack patterns, but the residual risk is not zero. Probe: a red-team query sequence run against a staging board; monitor the audit log for the patterns the attack produces. See Anti-overfit.

  • Key-person and seed-leak scenarios. (Severity: high, architectural, deferred to open questions) The held-out secret seed is held by a single external co-steward. If the co-steward is compromised or the seed leaks, the held-out is burned. Probe: re-materialize the held-out under a new secret seed on a fixed schedule and check that scores are stable across seeds; document the co-steward succession plan before the board goes live.

  • Sim-to-real confound. (Severity: medium) A model can score well on synthetic and poorly on real captures; the gap column is the mitigation, but only if the real subset is large enough (section 3). Probe: the gap-stability bootstrap in section 5.

  • Threshold drift. (Severity: medium) A sim-to-real gap threshold tuned per submission re-imports self-grading. Probe: the frozen-anchor read test in section 5 asserts the threshold and OOD-avg anchors the scorer loads are byte-for-byte equal to the pinned release artifacts, so no submission sees a different threshold.

  • Operator self-grading. (Severity: high) If the operator’s model is ever scored without an independent re-score, neutrality is broken. Probe: a contract test that every operator submission carries a pre_registration_id and a matching ReScoringLog entry; absence fails the build.

7. Synthesis

Load-bearing claims supported. The results-only board, the secret-seed holdout, and the anti-overfit guards together make a posted rank difficult to game and impossible to self-grade by the operator. The ring-fence instruments (pre-registration, re-scoring log) are the right construct for neutrality, following the MLPerf and ImageNet precedents.

Gaps surfaced. (1) The real-capture subset may be too small for a stable gap estimate at 8,000 captures; the gap must ship with its CI and the subset must grow if the CI is wider than the threshold. (2) The sim-to-real gap threshold is proposed-contract and uncalibrated until the first real scores exist. (3) The held-out secret seed has a key-person concentration at a single co-steward; the succession plan is not yet documented. (4) Adversarial query extraction is a residual risk that is monitored, not eliminated.

Recommended actions. (1) Land the re-scoring equivalence test before the board goes live; it is the cheapest insurance against operator self-grading. (2) Freeze the sim-to-real gap threshold by a fixed rule once the first real-capture scores exist, and report the gap with its CI. (3) Document the co-steward succession and seed-rotation schedule. (4) Add a red-team query-extraction probe to the staging board before opening public submission.

Top three findings.

  1. (high) The operator’s model must be re-scored by an external co-steward on every release from day one; neutrality cannot be retrofitted, and the re-scoring equivalence test is the proposed first verification.

  2. (medium) The EMMA-REAL-OOD subset at roughly 8,000 captures may be too small for a stable sim-to-real gap; the gap must be reported with its bootstrap CI, and the subset grown until the CI half-width is below the threshold.

  3. (medium) The held-out secret seed concentrates on a single co-steward; the succession plan and seed-rotation schedule must be documented before the board goes live.

References

  • Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Neutral governance formalized after launch; the re-scoring and submission-rules precedent.

  • Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” IJCV 2015, arXiv:1409.0575. Seeded with baselines; governance followed adoption.

  • Tramer, Zhang, Juels, Reiter, and Ristenpart, “Stealing Machine Learning Models via Prediction APIs,” USENIX Security Symposium 2016, arXiv:1609.02943. The adversarial query-extraction residual risk the anti-overfit guard monitors. (verify)

  • International Committee of Medical Journal Editors (ICMJE), “Clinical Trial Registration.” The pre-registration analogy for the operator seed-backbone declaration. (verify)

  • “RF-Analyzer,” 2026, arXiv:2605.04676. Sim-to-real generalization breaks in low-SNR and OOD regimes; the load-bearing caveat the gap column surfaces. (verify)

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Results-only evaluation under controlled shift.

See Also