CLI reference

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The emma CLI (command-line interface) is the evaluation harness. It regenerates datasets via frozen rfgen recipes, evaluates a frozen backbone across tasks under the OOD (out-of-distribution) protocol, and packages the prediction bundles a leaderboard submission needs. The per-command contract lives in API / CLI; this page is the worked command reference, with signatures, flags, runnable examples, and the output each command produces. Every command writes structured output (columnar text or JSON) so it pipes cleanly into scripts.

Install (planned)

pip install emma-benchmark

Until the package ships, the commands below are proposed and not runnable. The signatures match API / CLI so an implementer can turn each into a Typer command directly.

Command inventory

Command	Version	Maturity	Purpose
emma list-tasks	v0.1	proposed-contract	List tasks, pillars, metrics, OOD axes, and versions
emma download	v0.1	proposed-contract	Regenerate a split locally from a frozen rfgen recipe
emma eval	v0.1	proposed-contract	Evaluate a frozen backbone on a task under the OOD protocol
emma submit	v0.1	proposed-contract	Validate and submit prediction bundles for holdout scoring
emma score	v0.1	proposed-contract	Score predictions against a local dev split
emma release	v0.1	proposed-contract	Operator: cut a hashed data release and freeze anchors

---

emma list-tasks

List available tasks, their pillars, metrics, OOD axis, and release version.

Signature

emma list-tasks [--version V0_1] [--format table|json]

Arguments

Argument	Type	Default	Purpose
--version	ReleaseVersion value	v0.1	Filter tasks by release
--format	table | json	table	Output format; json is machine-readable for scripting

Example

$ emma list-tasks --version v0.1

ID            PILLAR         METRIC                      OOD AXIS              STATUS
E-LOC-AOA     localization   mean angular error (deg)    unseen environment    v0.1
E-LOC-LOS     localization   balanced accuracy / AUC     unseen environment    v0.1
E-ID-DRONE    identity       AUC                         unseen environment    v0.1
E-ID-FP       identity       top-1 / EER                 leave-one-unit-out    v0.1
E-ID-AMC      identity       accuracy (continuity)       unseen SNR regime     v0.1

E-ID-AMC (AMC is automatic modulation classification) is a continuity column: reported but excluded from the aggregated OOD average. See Task reference for the full per-task contract.

---

emma download

Regenerate a split locally from a frozen rfgen commit and seed range. The generator is the dataset; nothing large is downloaded as a blob. EMMA calls rfgen under the hood using the pinned RfgenRecipe; it never generates I/Q (in-phase and quadrature) itself. See Datasets and scenes.

Signature

emma download --split SPLIT [--tasks TASKS] [--out PATH] [--release RELEASE]

Arguments

Argument	Type	Default	Purpose
--split	Split value	required	Which split to regenerate (train or dev; holdout is rejected)
--tasks	comma-separated TaskID values	all v0.1 tasks	Tasks whose recipes to regenerate
--out	str	./scenes	Output directory for regenerated scenes
--release	ReleaseVersion value	v0.1	Release to resolve recipes from

Example

$ emma download --split dev --tasks E-LOC-AOA,E-ID-DRONE --out ./scenes

regenerating EMMA-LOC-v0.1 (dev) via rfgen@a91f3c2
  config : scenes/v0.1/urban-mimo-ura.yaml
  tasks  : E-LOC-AOA, E-ID-DRONE
  seeds  : 160001..200000 (40000 scenes)
  hash   : sha256:b7e3...f019
regenerating EMMA-DRONE-v0.1 (dev) via rfgen@a91f3c2
  ...
written : ./scenes/

The hash line is the content hash of the regenerated split (see Content hashing). Re-running with the same recipe reproduces the same scenes bit-for-bit.

---

emma eval

Evaluate a frozen backbone on one or more tasks under the OOD protocol, writing predictions, a metrics report, and a run manifest. Runs the full protocol by default: every non-training environment becomes a held-out fold. The readout head is re-fit per fold; the backbone weights are frozen.

Signature

emma eval --task TASK --model PATH --scenes PATH [--protocol PROTOCOL] [--out PATH]

Arguments

Argument	Type	Default	Purpose
--task	TaskID value	required	Task to evaluate
--model	str (path or identifier)	required	Backbone checkpoint or identifier
--scenes	str (directory)	required	Directory of regenerated scenes
--protocol	OOD protocol name	task default	Protocol to apply (leave_one_environment_out, leave_one_unit_out, leave_one_band_out)
--out	str	./runs/<task>	Output directory for predictions and metrics

Example

$ emma eval \
    --task E-LOC-AOA \
    --model ./checkpoints/emma-seed-v0.1 \
    --scenes ./scenes \
    --protocol leave_one_environment_out \
    --out ./runs/aoa

E-LOC-AOA  (leave-one-environment-out)
  train envs : E-URBAN
  eval  envs : E-RURAL
  mean angular error : 6.41 deg   (target: lower is better)
  E-RURAL MAE         : 7.02 deg
  E-URBAN  MAE         : 5.80 deg
  sim-to-real gap     : +4.1 pp  (vs EMMA-REAL-OOD)

predictions written : ./runs/aoa/E-LOC-AOA.parquet
metrics report       : ./runs/aoa/metrics.json
run manifest         : ./runs/aoa/run-manifest.json

MAE is mean angular error; pp is percentage points. The run manifest is a RunManifest pinning the (model, data, metric) triple so the run is reproducible from the output directory alone. The sim-to-real gap compares the synthetic score against the real-capture OOD subset; see Sim-to-real gap.

---

emma submit

Validate a prediction bundle and submit it to the leaderboard for holdout scoring. Each bundle is one Parquet file per task, schema-validated locally before upload. At v0.1 submission is results-only: the model itself never leaves your machine, only its predictions upload.

Signature

emma submit --leaderboard VERSION --predictions PATH --model NAME [--pre-registration ID]

Arguments

Argument	Type	Default	Purpose
--leaderboard	ReleaseVersion value	required	Leaderboard track
--predictions	str (directory)	required	Directory of prediction bundles
--model	str	required	Model display name
--pre-registration	str	None	Pre-registration id; required for operator (Superpose) submissions

Example

$ emma submit --leaderboard v0.1 --predictions ./runs/all --model "my-fm@v1"

validating 2 prediction bundle(s)...
  E-LOC-AOA   schema ok   hash sha256:a1b2...c3d4
  E-ID-DRONE  schema ok   hash sha256:e5f6...7a8b
submitting to leaderboard v0.1...
posted row : my-fm@v1
holdout scoring will run on the private split; results appear on the public board.

The holdout labels never leave the scoring server. See Prediction bundle for the per-task Parquet schema and Submission for the manifest contract.

---

emma score

Score a prediction bundle against a local dev split, producing a metrics report without contacting the leaderboard. Used for local validation before submission, and by ops to reproduce a posted number against a re-derived split.

Signature

emma score --task TASK --predictions PATH --scenes PATH [--out PATH]

Arguments

Argument	Type	Default	Purpose
--task	TaskID value	required	Task to score
--predictions	str (file)	required	Path to the prediction Parquet
--scenes	str (directory)	required	Directory of scenes with labels
--out	str	stdout	Output path for the metrics report

Example

$ emma score --task E-LOC-AOA \
    --predictions ./runs/aoa/E-LOC-AOA.parquet \
    --scenes ./scenes

scoring E-LOC-AOA against ./scenes (dev)
  mean angular error : 6.41 deg
  per-environment     : {E_RURAL: 7.02 deg, E_URBAN: 5.80 deg}
report written : stdout

---

emma release

Operator-only. Materialize a data release from one or more recipes, producing a DataReleaseManifest and content-hashing the output. Used by the release co-steward to cut a hashed release and freeze the normalization anchors a leaderboard track scores against.

Signature

emma release --recipes PATH [--release-id ID] [--out PATH]

Arguments

Argument	Type	Default	Purpose
--recipes	str (file or directory)	required	Recipe files to bundle
--release-id	str	auto-generated	Canonical release identifier
--out	str	./releases	Output directory for the release manifest and scenes

Example

$ emma release --recipes recipes/v0.1/ --release-id EMMA-v0.1-rc1 --out ./releases

materializing release from recipes/v0.1/
  recipes      : 5
  release id   : EMMA-v0.1-rc1
  scenes       : ./releases/scenes/
  manifest     : ./releases/EMMA-v0.1-rc1.json
  data hash    : sha256:c1d4...9a02
anchors frozen : normalization anchors recorded for leaderboard v0.1

The data hash is the content hash of the DataReleaseManifest, recorded on every leaderboard row as the data half of the auditable result triple. See Content hashing.

See Also

• API / CLI: the per-command contract this reference expands.

• Task reference: the task IDs, metrics, and OOD axes the CLI reads.

• Regenerate a dataset: the step-by-step procedure behind emma download.

• Evaluate a backbone: the procedure behind emma eval.

• Make a submission: the procedure behind emma submit.

• Schemas: the prediction bundle and submission contracts the CLI reads and writes.