Quickstart

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

The four-command path from a clean install to a local mean angular error on E-LOC-AOA, with no submission and nothing leaving your machine. This is the loop you run before touching the leaderboard: it confirms the install, the rfgen handoff, and the evaluation path end to end. The deeper, step-by-step version lives in Run locally.

Prerequisites

  • The emma CLI (command-line interface) is installed (see Install).

  • rfgen is installed and on your PATH, because emma download shells out to it.

The four commands

  1. Install the package.

    $ pip install emma-benchmark

  2. Regenerate a small dev split for E-LOC-AOA. The generator is the dataset, so this runs rfgen against the frozen recipe rather than downloading a blob.

    $ emma download --split dev --tasks E-LOC-AOA --out ./scenes

    The command prints the rfgen commit, the resolved configuration, the seed range, and the content hash. Keep the hash; it pins which scenes you scored against.

  3. Evaluate the seed backbone on E-LOC-AOA under its default OOD (out-of-distribution) axis. The backbone weights are frozen; only the readout head trains.

    $ emma eval \
    --task E-LOC-AOA \
    --model ./checkpoints/emma-seed-v0.1 \
    --scenes ./scenes \
    --out ./runs/local

  4. Read the metric from the console. The full report is in ./runs/local/metrics.json.

    E-LOC-AOA  (leave-one-environment-out)
  mean angular error : 6.41 deg   (target: lower is better)
  sim-to-real gap    : +4.1 pp    (vs EMMA-REAL-OOD)

  5. Score the same predictions locally without re-running the backbone, to confirm the number is reproducible.

    $ emma score --task E-LOC-AOA \
    --predictions ./runs/local/E-LOC-AOA.parquet \
    --scenes ./scenes

Expected result

./scenes/ holds a regenerated dev split, ./runs/local/ holds a prediction Parquet, a metrics report, and a run manifest, and the console shows a mean angular error. MAE is mean angular error; pp is percentage points. Nothing left your machine. When this loop is green, the next step is First submission.

See Also