First submission

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

The path from a green local evaluation to a v0.1 leaderboard submission. At v0.1 submission is results-only: the model itself never leaves your machine, only its predictions upload, and EMMA (Electromagnetic Multi-task Model Assessment) scores them against the private holdout. The deeper, step-by-step version lives in Make a submission.

Prerequisites

  • The emma CLI (command-line interface) is installed and you have completed Quickstart.

  • You have model provenance ready: a training-config hash, a code commit, and a weights URI (uniform resource identifier). A submission without provenance is rejected (see Submission).

Steps

  1. Produce a prediction bundle for each v0.1 scored task. Run emma eval once per task; each run writes a schema-validated Parquet file. The v0.1 scored tasks are E-LOC-AOA, E-LOC-LOS, E-ID-DRONE, and E-ID-FP. (E-ID-AMC is a continuity column and excluded from the aggregate.)

    $ emma eval --task E-LOC-AOA  --model ./checkpoints/my-fm --scenes ./scenes --out ./runs/all
$ emma eval --task E-LOC-LOS  --model ./checkpoints/my-fm --scenes ./scenes --out ./runs/all
$ emma eval --task E-ID-DRONE --model ./checkpoints/my-fm --scenes ./scenes --out ./runs/all
$ emma eval --task E-ID-FP    --model ./checkpoints/my-fm --scenes ./scenes --out ./runs/all

    Each Parquet is keyed by scene_id and carries the prediction columns for its task family. See Prediction bundle for the per-family columns.

  2. Submit the bundle directory to the v0.1 board. The CLI validates every bundle’s schema before upload; a schema mismatch stops the submission and no score is computed.

    $ emma submit --leaderboard v0.1 --predictions ./runs/all --model "my-fm@v1"

    validating 4 prediction bundle(s)...
  E-LOC-AOA   schema ok   hash sha256:a1b2...c3d4
  E-LOC-LOS   schema ok   hash sha256:9f8e...7d6c
  E-ID-DRONE  schema ok   hash sha256:e5f6...7a8b
  E-ID-FP     schema ok   hash sha256:1a2b...3c4d
submitting to leaderboard v0.1...
posted row : my-fm@v1

  3. Read the leaderboard row. EMMA scores your predictions against the private holdout and posts a row carrying the per-task metrics, the OOD average, the sim-to-real gap, and the prediction-bundle and data-release hashes.

    Model        Submitter   E-LOC-AOA (MAE)   OOD-avg   sim-to-real gap
my-fm@v1     team-alpha  5.1 deg           0.83      3.0 pp

    MAE is mean angular error; OOD-avg is the normalized transfer average across scored tasks; pp is percentage points. The hashes let any reviewer reproduce the number (see Reproduce a row).

Expected result

Your model appears on the public v0.1 board with its OOD average and sim-to-real gap. The model weights stayed on your machine; only the prediction Parquets uploaded. Operator (Superpose) submissions additionally require a pre-registration id, enforced by the submission validator.

See Also