Run manifest¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

A run manifest is the auditable provenance record for one evaluation run. It pins the (model, data, metric) triple that produced a set of scores: which model provenance was evaluated, which data release was scored, which prediction bundles were scored, which metrics resulted, and under which seed and harness version. The serialized form is the RunManifest Pydantic model. It is written by Evaluator on every run and recorded alongside the EvalRun result, so a score is reproducible from the manifest alone rather than self-reported.

The run manifest is the local-evaluation counterpart to the LeaderboardRow audit hashes. A leaderboard row records the data-release hash and the per-task metric values a reviewer needs to re-derive a posted score; a run manifest additionally records the per-task prediction-bundle hashes (the model half of the result triple) for any run, posted or not, including a re-scoring audit run by an external co-steward (see Neutrality).

Schema fields¶

Field	Type	Purpose
`model_provenance`	`str`	Training config hash, code commit, or checkpoint reference, matching the submission’s `model_provenance`.
`data_release_hash`	`str`	Content hash of the DataReleaseManifest scored.
`prediction_bundle_hashes`	`dict[str, str]`	Per-task PredictionBundle hashes, keyed by TaskID value. The model half of the result triple; surfaces for audit where the LeaderboardRow does not carry it.
`task_metrics`	`dict[str, float]`	Per-task metric values from the run, keyed by TaskID value.
`seed`	`int`	Seed used for readout-head fitting and fold construction.
`harness_version`	`str`	EMMA (Electromagnetic Multi-task Model Assessment) harness version that produced the run.
`timestamp`	`str`	ISO 8601 write timestamp, recorded when the manifest is serialized.

Note

The manifest is produced by the harness, not by the submitter. Because Evaluator writes it on every run, the task_metrics it records are the values the scoring code actually computed, not values the submitter typed in. A re-scoring run by an external co-steward writes its own manifest; matching task_metrics across the two manifests is the evidence that the operator’s number and the independent number agree.

Reproducibility contract¶

Given a run manifest, a reviewer can re-run the same evaluation: fetch the model by its provenance, re-derive the data release by its hash (see Content hashing), and re-execute the harness at the recorded harness_version and seed. Deterministic scoring means the re-run reproduces the task_metrics bit-for-bit. The seed field matters because the readout head is re-fit per fold, and fold construction depends on it; pinning the seed makes the lightweight-head fit reproducible across runs.

Example¶

{
  "model_provenance": "sha256:9f2c...1e04 code=b71d4a8",
  "data_release_hash": "sha256:c1d4...9a02",
  "prediction_bundle_hashes": {
    "E_LOC_AOA": "sha256:a1b2...c3d4",
    "E_LOC_LOS": "sha256:b2c3...d4e5",
    "E_ID_DRONE": "sha256:e5f6...7a8b",
    "E_ID_FP": "sha256:f7a8...b9c0"
  },
  "task_metrics": {
    "E_LOC_AOA": 5.1,
    "E_LOC_LOS": 0.94,
    "E_ID_DRONE": 0.95,
    "E_ID_FP": 0.82
  },
  "seed": 200001,
  "harness_version": "emma-0.1.0",
  "timestamp": "2026-07-17T14:03:11Z"
}

References¶

Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Reproducible scoring with recorded provenance at benchmark scale; the run-manifest-as-contract precedent.
Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Documented splits and seeds as a reproducibility contract; the seed and data-hash fields.