Evaluate a backbone

Note

The commands on this page are proposed until the CLI (command-line interface) ships. The signatures match API / CLI; once src/emma/ lands the steps below run as written.

Evaluate a frozen backbone on one task under its OOD (out-of-distribution) protocol. The score is a transfer score, not in-distribution accuracy: the protocol holds out one environment, device, or band at a time, and the headline number is the mean over held-out folds.

Goal

Run emma eval on the E-LOC-AOA (angle-of-arrival) task, read the per-fold metrics report, and read the run manifest that pins the result.

Prerequisites

  • The emma CLI is installed.

  • You have regenerated a dev split into ./scenes (see Regenerate a dataset).

  • You have a backbone checkpoint at a local path or a resolvable identifier. The backbone must satisfy the FrozenBackbone protocol: weights are frozen, only the readout head trains.

Steps

  1. Confirm the task’s metric and OOD axis. The axis is part of the task contract; E-LOC-AOA uses leave-one-environment-out and scores MeanAngularError.

    $ emma list-tasks --version v0.1

  2. Run the evaluation. With no --protocol given, the task’s default axis applies. The CLI fits a ReadoutHead per fold, scores each held-out environment, and averages.

    $ emma eval \
    --task E-LOC-AOA \
    --model ./checkpoints/emma-seed-v0.1 \
    --scenes ./scenes \
    --out ./runs/aoa

  3. Read the metrics report. The console block shows the train and eval environments, the mean angular error (MAE), and the sim-to-real gap against the real-capture OOD subset.

    E-LOC-AOA  (leave-one-environment-out)
  train envs : E-URBAN
  eval  envs : E-RURAL
  mean angular error : 6.41 deg   (target: lower is better)
  E-RURAL MAE         : 7.02 deg
  E-URBAN  MAE         : 5.80 deg
  sim-to-real gap     : +4.1 pp  (vs EMMA-REAL-OOD)

    pp is percentage points. The full metrics are also written to ./runs/aoa/metrics.json.

  4. Read the run manifest. ./runs/aoa/run-manifest.json is a RunManifest recording the model provenance, the data-release hash, the per-task metric values, the seed, and the harness version. It is the artifact a reviewer needs to reproduce this run (see Reproduce a row).

Expected result

./runs/aoa/ contains the prediction Parquet (E-LOC-AOA.parquet), a metrics report, and a run manifest. The mean angular error and the sim-to-real gap are the two numbers to read first; the per-environment values show whether the score is dominated by one outlier environment.

See Also