Make a submission

Note

The commands on this page are proposed until the CLI (command-line interface) ships. The signatures match API / CLI; once src/emma/ lands the steps below run as written.

Produce prediction bundles for the v0.1 tasks and submit them to the leaderboard for holdout scoring. At v0.1 submission is results-only: the model itself never leaves your machine, only its predictions upload, and EMMA (Electromagnetic Multi-task Model Assessment) scores them against the private holdout.

Goal

Generate one prediction Parquet per scored task, validate and submit them, and read the resulting leaderboard row.

Prerequisites

  • The emma CLI is installed.

  • You have regenerated the dev split and run at least one evaluation (see Evaluate a backbone).

  • You have model provenance ready: a training-config hash, a code commit, and a weights URI (uniform resource identifier). A submission without provenance is rejected (see Submission).

Steps

  1. Produce a prediction bundle for each task you intend to submit. Run emma eval once per task; each run writes a schema-validated Parquet file. The v0.1 scored tasks are E-LOC-AOA, E-LOC-LOS, E-ID-DRONE, and E-ID-FP. (E-ID-AMC is a continuity column and excluded from the aggregate.)

    $ emma eval --task E-LOC-AOA --model ./checkpoints/my-fm --scenes ./scenes --out ./runs/all
$ emma eval --task E-ID-DRONE --model ./checkpoints/my-fm --scenes ./scenes --out ./runs/all

    Each Parquet is keyed by scene_id and carries the prediction columns for its task family. See Prediction bundle for the per-family columns.

  2. Submit the bundle directory to the v0.1 board. The CLI validates every bundle’s schema before upload; a schema mismatch stops the submission and no score is computed.

    $ emma submit --leaderboard v0.1 --predictions ./runs/all --model "my-fm@v1"

    validating 2 prediction bundle(s)...
  E-LOC-AOA   schema ok   hash sha256:a1b2...c3d4
  E-ID-DRONE  schema ok   hash sha256:e5f6...7a8b
submitting to leaderboard v0.1...
posted row : my-fm@v1

  3. Read the leaderboard row. EMMA scores your predictions against the private holdout and posts a LeaderboardRow carrying the per-task metrics, the OOD (out-of-distribution) average, the sim-to-real gap, and the prediction-bundle and data-release hashes.

    Model        Submitter   E-LOC-AOA (MAE)   E-ID-DRONE (AUC)   OOD-avg   sim-to-real gap
my-fm@v1     team-alpha  5.1 deg           0.95               0.83      3.0 pp

    AUC is area under the ROC (receiver operating characteristic) curve; MAE is mean angular error; pp is percentage points.

Expected result

Your model appears on the public v0.1 board with its OOD average and sim-to-real gap. The model weights stayed on your machine; only the prediction Parquets uploaded. The posted row’s hashes let any reviewer reproduce the number (see Reproduce a row).

See Also

  • Submission: the manifest the CLI sends, including the model-provenance requirement.

  • Prediction bundle: the per-task Parquet format each bundle uses.

  • Leaderboard: the reporting surface and the sim-to-real gap column.

  • Reproduce a row: how a reviewer re-derives the number your row posts.

  • CLI reference: the full emma submit command reference.