CLI

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

Command surface for the emma evaluation harness. The CLI is a Typer application with Hydra-style composition for configuration. The worked examples live in CLI reference; this page is the per-command contract. Every command writes structured output (JSON or columnar text) so it can be piped into scripts.

emma list-tasks

List available tasks, their pillars, metrics, OOD (out-of-distribution) axis, and release version.

Signature

emma list-tasks [--version V0_1] [--format table|json]

Arguments

Argument	Type	Default	Purpose
--version	ReleaseVersion value	v0.1	Filter tasks by release
--format	table | json	table	Output format

Example

$ emma list-tasks --version v0.1
ID            PILLAR         METRIC                      OOD                    STATUS
E-LOC-AOA     localization   mean angular error (deg)    unseen environment     v0.1
E-LOC-LOS     localization   balanced accuracy / AUC     unseen environment     v0.1
E-ID-DRONE    identity       AUC                         unseen drone model     v0.1
E-ID-FP       identity       top-1 / EER                 leave-one-unit-out     v0.1
E-ID-AMC      identity       accuracy (continuity)       unseen SNR regime      v0.1

---

emma download

Regenerate a split locally from a frozen rfgen commit and seed range. The generator is the dataset; nothing large is downloaded as a blob. EMMA calls rfgen under the hood using the pinned RfgenRecipe; it never generates I/Q (in-phase and quadrature) itself.

Signature

emma download --split SPLIT [--tasks TASKS] [--out PATH] [--release RELEASE]

Arguments

Argument	Type	Default	Purpose
--split	Split value	required	Which split to regenerate (train or dev; holdout is rejected)
--tasks	comma-separated TaskID values	all v0.1 tasks	Tasks whose recipes to regenerate
--out	str	./scenes	Output directory for regenerated scenes
--release	ReleaseVersion value	v0.1	Release to resolve recipes from

Example

emma download --split dev --tasks E-LOC-AOA,E-ID-DRONE --out ./scenes

---

emma eval

Evaluate a backbone on one or more tasks under the OOD protocol, writing predictions and a metrics report. Runs the full protocol by default (all non-training environments as held-out).

Signature

emma eval --task TASK --model PATH --scenes PATH [--protocol PROTOCOL] [--out PATH]

Arguments

Argument	Type	Default	Purpose
--task	TaskID value	required	Task to evaluate
--model	str (path or identifier)	required	Backbone checkpoint or identifier
--scenes	str (directory)	required	Directory of regenerated scenes
--protocol	OODProtocol name	task default	Protocol to apply (leave_one_environment_out, leave_one_unit_out, leave_one_band_out)
--out	str	./runs/<task>	Output directory for predictions and metrics

Example

emma eval \
  --task E-LOC-AOA \
  --model ./checkpoints/emma-seed-v0.1 \
  --scenes ./scenes \
  --protocol leave_one_environment_out \
  --out ./runs/aoa

E-LOC-AOA  (leave-one-environment-out)
  train envs : E-URBAN
  eval  envs : E-RURAL
  mean angular error : 6.41 deg   (target: lower is better)
  E-RURAL MAE         : 7.02 deg
  sim-to-real gap     : +4.1 pp  (vs EMMA-REAL-OOD)

---

emma submit

Validate a prediction bundle and submit it to the leaderboard for holdout scoring.

Signature

emma submit --leaderboard VERSION --predictions PATH --model NAME [--pre-registration ID]

Arguments

Argument	Type	Default	Purpose
--leaderboard	ReleaseVersion value	required	Leaderboard track
--predictions	str (directory)	required	Directory of prediction bundles
--model	str	required	Model display name
--pre-registration	str	None	Pre-registration id; required for operator submissions

Example

emma submit --leaderboard v0.1 --predictions ./runs/all --model "my-fm@v1"

The holdout labels never leave the scoring server. See Prediction file format for the per-task Parquet schema.

---

emma score

Score a prediction bundle against a local dev split, producing a metrics report without contacting the leaderboard. Used for local validation before submission.

Signature

emma score --task TASK --predictions PATH --scenes PATH [--out PATH]

Arguments

Argument	Type	Default	Purpose
--task	TaskID value	required	Task to score
--predictions	str (file)	required	Path to the prediction Parquet
--scenes	str (directory)	required	Directory of scenes with labels
--out	str	stdout	Output path for the metrics report

Example

emma score --task E-LOC-AOA --predictions ./runs/aoa/E-LOC-AOA.parquet --scenes ./scenes

---

emma release

Materialize a data release from one or more recipes, producing a DataReleaseManifest and content-hashing the output. Operator-only; used by the release co-steward.

Signature

emma release --recipes PATH [--release-id ID] [--out PATH]

Arguments

Argument	Type	Default	Purpose
--recipes	str (file or directory)	required	Recipe files to bundle
--release-id	str	auto-generated	Canonical release identifier
--out	str	./releases	Output directory for the release manifest and scenes

Example

emma release --recipes recipes/v0.1/ --release-id EMMA-v0.1-rc1 --out ./releases

See Also

• CLI reference: worked examples and the install path.

• Schemas: the prediction bundle and submission contracts the CLI reads and writes.

• Harness: the evaluator and protocol runner the eval command drives.

• Tasks: the task and protocol contracts the eval and list-tasks commands operate on.

• Regenerate a dataset: the step-by-step procedure behind emma download.