Run locally

Note

The commands on this page are proposed until the CLI (command-line interface) ships. Once src/emma/ lands this is the shortest path from install to a metric on your machine.

Install the harness, regenerate a small dev split, and run one evaluation locally with no submission. This is the loop you run before touching the leaderboard: it confirms the install, the rfgen handoff, and the evaluation path end to end.

Goal

Go from a clean install to a local mean angular error on E-LOC-AOA, without contacting any scoring server.

Prerequisites

  • Python 3.10 or newer.

  • rfgen available on your machine; emma download shells out to it to regenerate scenes.

Steps

  1. Install the package.

    $ pip install emma-benchmark

    Confirm the CLI is on your PATH.

    $ emma list-tasks --version v0.1

  2. Regenerate a small dev split. Limit the task list to one task and point --out at a local directory. The generator is the dataset, so this step runs rfgen rather than downloading a blob.

    $ emma download --split dev --tasks E-LOC-AOA --out ./scenes

    The command prints the rfgen commit, the resolved configuration, the seed range, and the content hash. Keep the hash; it pins which scenes you scored against (see Content hashing).

  3. Run one evaluation locally. Use a backbone checkpoint you have on disk. With no --protocol given, the task’s default OOD (out-of-distribution) axis applies.

    $ emma eval \
    --task E-LOC-AOA \
    --model ./checkpoints/emma-seed-v0.1 \
    --scenes ./scenes \
    --out ./runs/local

  4. Read the metric. The console prints the mean angular error and the per-environment values; the full report is in ./runs/local/metrics.json.

    E-LOC-AOA  (leave-one-environment-out)
  mean angular error : 6.41 deg   (target: lower is better)

Expected result

./scenes/ holds a regenerated dev split, ./runs/local/ holds a prediction Parquet, a metrics report, and a run manifest, and the console shows a mean angular error. Nothing left your machine. When this loop is green, the next step is Make a submission.

See Also