Content hashing

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The reproducibility contract. A benchmark result is credible only if a third party can re-derive the data and re-score the model. EMMA (Electromagnetic Multi-task Model Assessment) achieves this by making the generator, not a static file, the test set: every scene is a deterministic function of a frozen rfgen commit, a resolved configuration, and a seed. EMMA content-hashes that triple, so the public train and dev (development) splits are re-derivable by anyone, bit-for-bit, from the recipe alone, and the holdout split is the same mechanism with a secret seed. A leaderboard row is then an auditable triple of (model, data, metric), each carrying its own hash. The narrative motivation lives in Reproducibility.

The scene triple

A scene is fully determined by three inputs:

1. The rfgen commit (a git SHA, Secure Hash Algorithm digest).

2. The resolved rfgen configuration (the fully composed config, not just the top-level file).

3. The integer seed.

The DatasetRecipe pins this triple; the DataRelease materializes it. Two recipes with the same triple produce the same scenes; two recipes with different triples do not.

The content hash

EMMA computes a SHA-256 digest over the (commit, config, seed) triple. The canonical digest construction is:

\[ h = \mathrm{SHA\text{-}256}\!\bigl(\text{commit} \;\Vert\; \text{canonical\_config\_json} \;\Vert\; \text{seed}\bigr) \]

where \(\Vert\) is byte concatenation, canonical_config_json is the resolved config serialized to canonical JSON (stable key order, no comments, no whitespace variation), and the inputs are length-delimited so that two different triples cannot collide by concatenation ambiguity. The digest is recorded in the SceneManifest for each scene and aggregated in the DataReleaseManifest.

• Determinism. The digest is a pure function of the triple. Re-deriving the same recipe yields the same hash, bit-for-bit.

• Collision resistance. SHA-256 gives practical collision resistance; two distinct recipes hashing to the same value is computationally infeasible.

• Config canonicalization. Only the resolved config is hashed, so cosmetic edits to the source YAML (comments, key order in the file) that do not change the resolved config leave the hash unchanged.

Splits and the held-out secret

Split	Seed	Released
train	Public seed range.	Public, re-derivable from the recipe.
dev	Public seed range.	Public, re-derivable from the recipe.
holdout	Frozen secret seed held by the external co-steward.	Private, never released.

The holdout split uses the same content-hash mechanism as train and dev, but its seed range is a frozen secret. The data never exists on disk for anyone but the co-steward: it is re-materialized only inside the scoring path. Scores on it are produced by prediction submission at v0.1 (the submitter submits predictions; EMMA scores them) and by sandboxed code submission at v1.

The auditable result triple

A leaderboard row is not a self-reported number. It is a triple recorded on the LeaderboardRow:

1. Model. Identified by the submitter; the prediction-bundle hash from PredictionBundle pins exactly which outputs were scored.

2. Data. Identified by the data-release hash, pinning exactly which scenes were scored against.

3. Metric. The deterministic scoring code and its frozen normalization anchors (see Aggregation).

A reviewer can re-derive the data from the recipe, re-score the predictions with the published metric, and check the number. Provenance survives fixes: a corrected release gets a new hash, and the old release remains citable because the hash is part of the result triple.

Why not a static file

Static-file benchmarks fail in three predictable ways, all of which content-hashing addresses:

• Frozen label errors. RadioML’s publisher admits errata baked into a released file; there is no fix without invalidating every result. A content-hashed release fixes a bug by shipping a new hash, and the old hash remains auditable.

• Dying eval servers. When the scoring server goes down, the benchmark is over. Because the data is re-derivable and the metric is published code, anyone can re-score locally.

• Leaking test sets. A static held-out file can be downloaded and trained on. A held-out that is a secret seed cannot, because the data never exists on disk for anyone but the co-steward.

References

• Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Re-derivable, split-documented benchmark data as a contract.

• DeepSig, “RadioML datasets.” The publisher’s public note that RML2016 has known errata and is not used in DeepSig products; the frozen-label-error failure mode content hashing addresses. (verify URL)

• Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Reproducible scoring and auditable results at benchmark scale.

• NIST, “Secure Hash Standard (SHS),” FIPS PUB 180-4. SHA-256 digest construction. (verify)

See Also

• Reproducibility: the narrative version of the content-hash contract.

• Dataset recipe: the recipe schema the hash pins.

• Scene manifest: the per-scene hash record.

• Leaderboard row: the result triple in context.

• Aggregation: the frozen normalization anchors the metric half of the triple depends on.