Reproducibility

Pre-launch / design spec

The content-hash scheme and held-out protocol below are proposed for v0.1. The exact digest construction finalizes with the data-release tooling.

Overview

A benchmark result is credible only if a third party can re-derive the data and re-score the model. EMMA (Electromagnetic Multi-task Model Assessment) achieves this by making the generator, not a static file, the test set. Every scene is a deterministic function of a frozen rfgen commit, a resolved configuration, and a seed. EMMA content-hashes that triple, so the public train and dev splits are re-derivable by anyone, bit-for-bit, from the recipe alone. The held-out split is the same mechanism with a secret seed. A leaderboard row is then an auditable triple of (model, data, metric), with the model and data each pinned by a hash.

Place In The System

Reproducibility spans the data layer and the leaderboard. At the data layer, a DatasetRecipe pins the (commit, config, seed) triple and a DataRelease materializes it. At the leaderboard layer, a LeaderboardRow carries the data-release hash and the per-task scores, and links its RunManifest for the per-task prediction-bundle hashes. See Datasets and rfgen and Leaderboard.

Boundaries

  • Owned by EMMA: the recipe schema, the content-hash construction, the data-release manifest, and the result-triple recorded on the leaderboard.

  • Owned by rfgen: determinism of scene generation from (commit, config, seed). EMMA depends on rfgen being deterministic; it does not implement that determinism itself.

Data Flow

(commit, config, seed) ──► content hash ──► DataRelease
        │                                         │
        ▼                                         ▼
  re-derivable by anyone            train / dev public, holdout secret
                                                    │
   prediction bundle (model outputs)               │
            │                                      │
            ▼                                      ▼
   prediction-bundle hash ──► RunManifest    LeaderboardRow
                              (prediction-   (data-release hash,
                               bundle hashes,  per-task scores)
                               one per task)
                                    │               ▲
                                    └─── links ─────┘
                                               │
                                               ▼
                                     auditable result triple

The Content Hash

A scene is fully determined by a triple:

  • the rfgen commit (a git SHA),

  • the resolved rfgen configuration (the fully composed config, not just the top-level file),

  • the integer seed.

EMMA computes a SHA-256 digest over this triple. Two recipes with the same hash produce the same scenes; two recipes with different hashes do not. The hash is recorded in the SceneManifest for each scene and aggregated in the DataReleaseManifest. The exact digest construction is documented at Content hashing.

Splits and the Held-Out Secret

Split

Purpose

Released

train

Pretraining and fine-tuning

public, re-derivable from the recipe

dev

Local evaluation and public proxies

public, re-derivable from the recipe

holdout

Official leaderboard scoring

private, never released

The holdout split uses the same content-hash mechanism as train and dev, but its seed range is a frozen secret held by the external co-steward. Scores on it are produced by prediction submission at v0.1 (you submit predictions; EMMA scores them) and by sandboxed code submission at v1. Superpose’s own model is re-scored on the holdout by a third party on every release. See Neutrality.

The Auditable Result Triple

A leaderboard row is not a self-reported number. It is a triple:

  1. Model: identified by the submitter, with the prediction-bundle hash pinning exactly what outputs were scored.

  2. Data: identified by the data-release hash, pinning exactly which scenes were scored against.

  3. Metric: the deterministic scoring code and its frozen normalization anchors.

The LeaderboardRow carries the data-release hash and the per-task scores; its linked RunManifest carries the per-task prediction-bundle hashes (one per task, from the PredictionBundle records). A reviewer can re-derive the data from the recipe, re-score the predictions with the published metric, and check the number.

Why Not a Static File

Static-file benchmarks fail in three predictable ways, all of which content-hashing addresses:

  • Frozen label errors. RadioML’s publisher admits errata baked into a released file; there is no fix without invalidating every result. A content-hashed release fixes a bug by shipping a new hash, and provenance is preserved because the old hash remains auditable.

  • Dying eval servers. When the scoring server goes down, the benchmark is over. Because the data is re-derivable and the metric is published code, anyone can re-score locally.

  • Leaking test sets. A static held-out file can be downloaded and trained on. A held-out that is a secret seed cannot, because the data never exists on disk for anyone but the co-steward.

Minimal Example

# re-derive a public split from its recipe (no download of a static file needed)
emma data derive --recipe EMMA-LOC-v0.1 --split dev --out ./scenes/

# verify a released data bundle matches its content hash
emma data verify --release EMMA-LOC-v0.1 --bundle ./scenes/

Design Notes

  • The generator is the contract. Reproducibility does not depend on a hosted archive. It depends on rfgen being deterministic and on the recipe pinning the triple.

  • Provenance survives fixes. A corrected release gets a new hash. The old release remains citable and its results remain comparable to each other, because the hash is part of the result triple.

  • The held-out is a secret seed, not a secret file. This is the property that makes the leaderboard score resilient to leakage without requiring a permanently online trusted server.

References

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Re-derivable, split-documented benchmark data as a contract.

  • DeepSig, “RadioML datasets.” The publisher’s public note that RML2016 has known errata and is not used in DeepSig products; the frozen-label-error failure mode content hashing addresses.

  • Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Reproducible scoring and auditable results at benchmark scale.

See Also