Determinism¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The determinism guarantees behind every EMMA (Electromagnetic Multi-task Model Assessment) number. A benchmark score is credible only if re-running the same evaluation reproduces the same result. EMMA pins two determinism surfaces: scene regeneration, where a frozen rfgen commit, resolved configuration, and seed regenerate the same scene byte-for-byte across processes; and result reproduction, where a leaderboard row is reproducible from its run manifest, the (model, data, metric) triple. This page states what is and is not deterministic, and where EMMA intervenes to close the gap. The content-hash construction that underpins both surfaces is audited in Content hashing; the run-level provenance contract lives in Run manifest.

Scene regeneration is byte-identical¶

A scene is a pure function of three inputs: the rfgen commit (a git SHA, Secure Hash Algorithm digest), the resolved rfgen configuration (the fully composed config, not just the top-level file), and the integer seed. The DatasetRecipe pins this triple, and two recipes with the same triple produce the same scenes.

Across processes. Re-deriving the same recipe in a different process, on a different machine, or at a different time yields the same generated I/Q (in-phase and quadrature) and the same metadata, byte-for-byte.
Verified by content hash. The SceneManifest records a content_hash over the generated bytes. A re-derivation that produces a different hash flags drift (a recipe pinned to the wrong commit, a config that did not fully resolve, or a generator bug), not a nondeterminism. See Content hashing.
Config canonicalization. Only the resolved config enters the hash. Cosmetic edits to the source YAML (comments, file-level key order) that do not change the resolved config leave the hash unchanged, so the determinism contract is over semantic content, not textual layout.

The determinism of the underlying rfgen generator is rfgen’s contract; EMMA consumes it. Where rfgen’s own validation reports assert determinism, this page cites them; where they do not, the gap is named in Datasets validation.

A leaderboard row is reproducible from its run manifest¶

A posted score is not a self-reported number. The auditable result triple spans a LeaderboardRow and its linked RunManifest: the row carries the data (via the data_release_hash) and the metric (via the per-task scores and the frozen normalization anchors), and the model half (the per-task prediction-bundle hashes) is on the linked RunManifest. A RunManifest records the same triple for any run, plus the seed and harness_version that make the lightweight readout-head fit reproducible.

Given a run manifest, a reviewer re-runs the same evaluation: fetch the model by its provenance, re-derive the data release by its hash, and re-execute the harness at the recorded harness_version and seed. Deterministic scoring means the re-run reproduces the task_metrics bit-for-bit. The seed matters because the readout head is re-fit per fold and fold construction depends on it; pinning the seed makes the head fit reproducible across runs. See Run manifest for the field contract.

What is deterministic¶

Surface	Guarantee	Mechanism
Scene bytes	Re-derivation is byte-identical across processes	Pinned (commit, resolved config, seed) plus content hash
Scene manifest	Re-derivation yields the same `content_hash`	SHA-256 over canonical scene content
Data release hash	Re-serialization is stable; tampering changes the hash	Canonical JSON serialization, length-delimited concatenation
Per-task scores	A re-run at the recorded seed and version reproduces the metric	Pinned seed, pinned harness version, deterministic metric code
Readout-head fit	The fold split and head weights are reproducible	Pinned seed for fold construction and head fitting

What is not deterministic, and how EMMA closes the gap¶

Floating-point arithmetic is not associative. A reduction whose summation order varies across runs can produce a different last-bit result, and a different last-bit result can change a rounded score. The risk is real for distributed executors, GPU (graphics processing unit) kernels with nondeterministic atomic operations, and any metric that sums over a sharded prediction set.

Reduction order is pinned. EMMA computes every metric reduction in a fixed order: the prediction rows enter the metric in a stable sort (by scene_id), and the reduction proceeds in that order rather than in whatever order a distributed executor happens to deliver. The same inputs in the same order yield the same floating-point result on the same hardware and library versions.
Library determinism is delegated. torchmetrics, torch, scipy, and numpy reductions are deterministic given fixed input order on a fixed platform, but GPU nondeterminism (convolution algorithm selection, atomic-add ordering) can still vary. EMMA pins the platform and the library versions in the run manifest’s harness_version, and flags any metric that cannot be made deterministic on the target platform as an open issue rather than asserting reproducibility it cannot guarantee.
Rounding is explicit. Posted scores are rounded for display, but the manifest records the unrounded task_metrics. A re-run compares unrounded values; the rounded leaderboard number is a presentation of the unrounded recorded value.

Note

Cross-platform bit-identity (the same score on a different GPU architecture or a different BLAS (Basic Linear Algebra Subprograms) backend) is a stronger claim than same-process reproducibility. EMMA guarantees same-process and same-platform reproducibility from the manifest; cross-platform agreement is asserted only up to floating-point tolerance, and the re-scoring equivalence test in Leaderboard validation uses a relative tolerance of 1e-6 rather than exact equality for exactly this reason.

References¶

NIST, “Secure Hash Standard (SHS),” FIPS PUB 180-4. The SHA-256 digest construction behind the content-hash determinism contract. (verify)
Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Documented splits and seeds as a reproducibility contract; the seed and data-hash fields. (verify)
Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Reproducible scoring with recorded provenance at benchmark scale; the run-manifest-as-contract precedent.