Reproduce a leaderboard row

Note

The commands on this page are proposed until the CLI (command-line interface) ships. The signatures match API / CLI; once src/emma/ lands the steps below run as written.

Reproduce a published leaderboard row from its provenance. A row is an auditable (model, data, metric) triple: the prediction-bundle hash pins which outputs were scored, the data-release hash pins which scenes were scored against, and the published scoring code pins the metric. Given the row’s run manifest, a reviewer can re-derive the data, re-score the predictions, and check the number.

Goal

Take a published row, regenerate the exact dev split it was scored on, and re-score the prediction bundle locally so the reproduced metric matches the posted one.

Prerequisites

  • The emma CLI is installed.

  • You have the RunManifest for the row (or the row’s public fields: the data-release hash and the per-task scores). Every row links its manifest; see Run manifest.

  • You have the prediction Parquet for the row, available from the leaderboard artifact store or the submitter.

Steps

  1. Read the row’s hashes. The run manifest carries data_release_hash (the data half of the triple) and a task_metrics block (the result). The prediction-bundle hashes live on the linked run manifest (RunManifest.prediction_bundle_hashes), one per task.

    data_release_hash : sha256:c1d4...9a02
prediction bundle  : sha256:a1b2...c3d4  (E-LOC-AOA)
task_metrics       : {E_LOC_AOA: 5.1}

  2. Regenerate the exact split. The data-release hash resolves to the pinned rfgen commit, resolved configuration, and seed range. emma download re-derives the scenes bit-for-bit; the printed hash must match data_release_hash.

    $ emma download --split dev --tasks E-LOC-AOA --out ./scenes

    hash : sha256:c1d4...9a02

    A mismatch means the recipe changed; re-derive from the manifest’s recipe, not the latest registry (see Content hashing).

  3. Re-score the prediction bundle against the regenerated split. emma score runs the same metric the leaderboard scoring path uses, without contacting the server.

    $ emma score --task E-LOC-AOA \
    --predictions ./E-LOC-AOA.parquet \
    --scenes ./scenes

    scoring E-LOC-AOA against ./scenes (dev)
  mean angular error : 5.1 deg

  4. Compare the reproduced metric to the posted one. Deterministic scoring means a re-run over the same data and predictions reproduces the task_metrics bit-for-bit. A mismatch flags a bug in the data, the predictions, or the scoring code; it is never silently tolerated.

Expected result

The reproduced mean angular error matches the row’s posted value, and the regenerated split’s hash matches the row’s data_release_hash. The result is now independently verified.

See Also