CLI reference¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The emma CLI (command-line interface) is the evaluation harness. It regenerates datasets via frozen rfgen recipes, evaluates a frozen backbone across tasks under the OOD (out-of-distribution) protocol, and packages the prediction bundles a leaderboard submission needs. The per-command contract lives in API / CLI; this page is the worked command reference, with signatures, flags, runnable examples, and the output each command produces. Every command writes structured output (columnar text or JSON) so it pipes cleanly into scripts.

Install (planned)¶

pip install emma-benchmark

Until the package ships, the commands below are proposed and not runnable. The signatures match API / CLI so an implementer can turn each into a Typer command directly.

Command inventory¶

Command	Version	Maturity	Purpose
`emma list-tasks`	v0.1	`proposed-contract`	List tasks, pillars, metrics, OOD axes, and versions
`emma download`	v0.1	`proposed-contract`	Regenerate a split locally from a frozen rfgen recipe
`emma eval`	v0.1	`proposed-contract`	Evaluate a frozen backbone on a task under the OOD protocol
`emma submit`	v0.1	`proposed-contract`	Validate and submit prediction bundles for holdout scoring
`emma score`	v0.1	`proposed-contract`	Score predictions against a local dev split
`emma release`	v0.1	`proposed-contract`	Operator: cut a hashed data release and freeze anchors

`emma list-tasks`¶

List available tasks, their pillars, metrics, OOD axis, and release version.

Signature¶

emma list-tasks [--version V0_1] [--format table|json]

Arguments¶

Argument	Type	Default	Purpose
`--version`	ReleaseVersion value	`v0.1`	Filter tasks by release
`--format`	`table \| json`	`table`	Output format; `json` is machine-readable for scripting

Example¶

$ emma list-tasks --version v0.1

ID            PILLAR         METRIC                      OOD AXIS              STATUS
E-LOC-AOA     localization   mean angular error (deg)    unseen environment    v0.1
E-LOC-LOS     localization   balanced accuracy / AUC     unseen environment    v0.1
E-ID-DRONE    identity       AUC                         unseen environment    v0.1
E-ID-FP       identity       top-1 / EER                 leave-one-unit-out    v0.1
E-ID-AMC      identity       accuracy (continuity)       unseen SNR regime     v0.1

E-ID-AMC (AMC is automatic modulation classification) is a continuity column: reported but excluded from the aggregated OOD average. See Task reference for the full per-task contract.

`emma download`¶

Regenerate a split locally from a frozen rfgen commit and seed range. The generator is the dataset; nothing large is downloaded as a blob. EMMA calls rfgen under the hood using the pinned RfgenRecipe; it never generates I/Q (in-phase and quadrature) itself. See Datasets and scenes.

Signature¶

emma download --split SPLIT [--tasks TASKS] [--out PATH] [--release RELEASE]

Arguments¶

Argument	Type	Default	Purpose
`--split`	Split value	required	Which split to regenerate (`train` or `dev`; `holdout` is rejected)
`--tasks`	comma-separated TaskID values	all v0.1 tasks	Tasks whose recipes to regenerate
`--out`	`str`	`./scenes`	Output directory for regenerated scenes
`--release`	ReleaseVersion value	`v0.1`	Release to resolve recipes from

Example¶

$ emma download --split dev --tasks E-LOC-AOA,E-ID-DRONE --out ./scenes

regenerating EMMA-LOC-v0.1 (dev) via rfgen@a91f3c2
  config : scenes/v0.1/urban-mimo-ura.yaml
  tasks  : E-LOC-AOA, E-ID-DRONE
  seeds  : 160001..200000 (40000 scenes)
  hash   : sha256:b7e3...f019
regenerating EMMA-DRONE-v0.1 (dev) via rfgen@a91f3c2
  ...
written : ./scenes/

The hash line is the content hash of the regenerated split (see Content hashing). Re-running with the same recipe reproduces the same scenes bit-for-bit.

`emma eval`¶

Evaluate a frozen backbone on one or more tasks under the OOD protocol, writing predictions, a metrics report, and a run manifest. Runs the full protocol by default: every non-training environment becomes a held-out fold. The readout head is re-fit per fold; the backbone weights are frozen.

Signature¶

emma eval --task TASK --model PATH --scenes PATH [--protocol PROTOCOL] [--out PATH]

Arguments¶

Argument	Type	Default	Purpose
`--task`	TaskID value	required	Task to evaluate
`--model`	`str` (path or identifier)	required	Backbone checkpoint or identifier
`--scenes`	`str` (directory)	required	Directory of regenerated scenes
`--protocol`	OOD protocol name	task default	Protocol to apply (`leave_one_environment_out`, `leave_one_unit_out`, `leave_one_band_out`)
`--out`	`str`	`./runs/<task>`	Output directory for predictions and metrics

Example¶

$ emma eval \
    --task E-LOC-AOA \
    --model ./checkpoints/emma-seed-v0.1 \
    --scenes ./scenes \
    --protocol leave_one_environment_out \
    --out ./runs/aoa

E-LOC-AOA  (leave-one-environment-out)
  train envs : E-URBAN
  eval  envs : E-RURAL
  mean angular error : 6.41 deg   (target: lower is better)
  E-RURAL MAE         : 7.02 deg
  E-URBAN  MAE         : 5.80 deg
  sim-to-real gap     : +4.1 pp  (vs EMMA-REAL-OOD)

predictions written : ./runs/aoa/E-LOC-AOA.parquet
metrics report       : ./runs/aoa/metrics.json
run manifest         : ./runs/aoa/run-manifest.json

MAE is mean angular error; pp is percentage points. The run manifest is a RunManifest pinning the (model, data, metric) triple so the run is reproducible from the output directory alone. The sim-to-real gap compares the synthetic score against the real-capture OOD subset; see Sim-to-real gap.

`emma submit`¶

Validate a prediction bundle and submit it to the leaderboard for holdout scoring. Each bundle is one Parquet file per task, schema-validated locally before upload. At v0.1 submission is results-only: the model itself never leaves your machine, only its predictions upload.

Signature¶

emma submit --leaderboard VERSION --predictions PATH --model NAME [--pre-registration ID]

Arguments¶

Argument	Type	Default	Purpose
`--leaderboard`	ReleaseVersion value	required	Leaderboard track
`--predictions`	`str` (directory)	required	Directory of prediction bundles
`--model`	`str`	required	Model display name
`--pre-registration`	`str`	`None`	Pre-registration id; required for operator (Superpose) submissions

Example¶

$ emma submit --leaderboard v0.1 --predictions ./runs/all --model "my-fm@v1"

validating 2 prediction bundle(s)...
  E-LOC-AOA   schema ok   hash sha256:a1b2...c3d4
  E-ID-DRONE  schema ok   hash sha256:e5f6...7a8b
submitting to leaderboard v0.1...
posted row : my-fm@v1
holdout scoring will run on the private split; results appear on the public board.

The holdout labels never leave the scoring server. See Prediction bundle for the per-task Parquet schema and Submission for the manifest contract.

`emma score`¶

Score a prediction bundle against a local dev split, producing a metrics report without contacting the leaderboard. Used for local validation before submission, and by ops to reproduce a posted number against a re-derived split.

Signature¶

emma score --task TASK --predictions PATH --scenes PATH [--out PATH]

Arguments¶

Argument	Type	Default	Purpose
`--task`	TaskID value	required	Task to score
`--predictions`	`str` (file)	required	Path to the prediction Parquet
`--scenes`	`str` (directory)	required	Directory of scenes with labels
`--out`	`str`	stdout	Output path for the metrics report

Example¶

$ emma score --task E-LOC-AOA \
    --predictions ./runs/aoa/E-LOC-AOA.parquet \
    --scenes ./scenes

scoring E-LOC-AOA against ./scenes (dev)
  mean angular error : 6.41 deg
  per-environment     : {E_RURAL: 7.02 deg, E_URBAN: 5.80 deg}
report written : stdout

`emma release`¶

Operator-only. Materialize a data release from one or more recipes, producing a DataReleaseManifest and content-hashing the output. Used by the release co-steward to cut a hashed release and freeze the normalization anchors a leaderboard track scores against.

Signature¶

emma release --recipes PATH [--release-id ID] [--out PATH]

Arguments¶

Argument	Type	Default	Purpose
`--recipes`	`str` (file or directory)	required	Recipe files to bundle
`--release-id`	`str`	auto-generated	Canonical release identifier
`--out`	`str`	`./releases`	Output directory for the release manifest and scenes

Example¶

$ emma release --recipes recipes/v0.1/ --release-id EMMA-v0.1-rc1 --out ./releases

materializing release from recipes/v0.1/
  recipes      : 5
  release id   : EMMA-v0.1-rc1
  scenes       : ./releases/scenes/
  manifest     : ./releases/EMMA-v0.1-rc1.json
  data hash    : sha256:c1d4...9a02
anchors frozen : normalization anchors recorded for leaderboard v0.1

The data hash is the content hash of the DataReleaseManifest, recorded on every leaderboard row as the data half of the auditable result triple. See Content hashing.

CLI reference¶

Install (planned)¶

Command inventory¶

emma list-tasks¶

Signature¶

Arguments¶

Example¶

emma download¶

Signature¶

Arguments¶

Example¶

emma eval¶

Signature¶

Arguments¶

Example¶

emma submit¶

Signature¶

Arguments¶

Example¶

emma score¶

Signature¶

Arguments¶

Example¶

emma release¶

Signature¶

Arguments¶

Example¶

See Also¶

`emma list-tasks`¶

`emma download`¶

`emma eval`¶

`emma submit`¶

`emma score`¶

`emma release`¶