CLI validation

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands. Once implementation exists, the planned tests below run against src/emma/.

Validation of the emma CLI (command-line interface). The CLI is an engineering layer, so this report carries the construct, literature, and methodology lenses in depth and keeps the mathematical-fidelity and empirical-realism lenses to the places they apply. The artifact under validation is the command surface in API / CLI and the worked reference in CLI reference; the planned tests run once src/emma/ lands.

The load-bearing question is narrow and specific: does every command faithfully expose the task, metric, and dataset-recipe contracts the harness owns, and does no command imply a capability the harness does not have?

1. Purpose and construct

The CLI is a thin wrapper over the Evaluator, the dataset loaders, and the leaderboard store. Its construct validity rests on four claims, each testable.

Claim CLI-C1: the command set matches the contract surface, one to one. Every documented command (list-tasks, download, eval, submit, score, release) maps to a concrete harness capability, and no undocumented command exists. A command that names an unimplemented capability is a construct violation: it advertises behavior the harness cannot back. The proposed contract test (section 5) introspects the Typer application and asserts the command set equals the documented set.

Claim CLI-C2: emma download regenerates, never downloads a blob. The command name is download, but the behavior is regeneration from a frozen rfgen commit, resolved configuration, and seed range (see Content hashing). This is the single sharpest construct claim, because the name and the behavior diverge on purpose: the generator, not a static file, is the dataset. A regression that introduced a static-blob download path would silently break the reproducibility contract. The test asserts that the command resolves a DatasetRecipe, calls rfgen (radio-frequency signal generation) and never generates I/Q (in-phase and quadrature) samples itself, and that the output content hash matches a re-derivation.

Claim CLI-C3: emma eval applies the OOD (out-of-distribution) protocol by default. No --protocol flag is required to score under transfer; the task’s pinned axis is the default. This makes the OOD protocol the path of least resistance rather than an opt-in, which is the methodological point of the benchmark. A regression that defaulted to in-distribution accuracy would invalidate every posted score. The test asserts that an eval invocation with no --protocol uses the task’s OODAxis and produces a per-fold mean, not a single in-distribution number.

Claim CLI-C4: emma submit is results-only at v0.1 and validates before upload. The model never leaves the submitter’s machine; only prediction Parquets upload, and each is schema-validated before any network call. The construct is “submit outputs, not code” at v0.1, hardening to “submit code” at v1. The test asserts that a schema-mismatched bundle is rejected locally with no upload attempted.

Omissions. The CLI does not expose a real-capture adapter registration command at v0.1; real-capture OOD subsets are wired through recipes, not the CLI. This is a documented scope limit, not a gap.

Downstream risk. A construct violation here propagates to the leaderboard undetected, because the CLI is the only entry point most users touch. The introspection test in section 5 is the primary guard.

2. Mathematical fidelity

The CLI does not implement math. It passes arguments to the harness and formats output. The one mathematical surface is the sim-to-real gap and per-fold mean printed by emma eval: those values come from OODAvg and SimToRealGap, not from CLI-side arithmetic. The test asserts the printed values equal the EvalRun values bit-for-bit, so the CLI never rounds or recomputes a metric.

3. Empirical realism

Not applicable to an engineering layer. The CLI does not generate or transform data; it shells out to rfgen and the harness. Empirical realism of the scenes is validated in Datasets validation.

4. Literature grounding

  • CLI framework. The harness uses Typer with Hydra-style composition for configuration. Both are established, maintained libraries; no hand-rolled argument parser ships. (verify pinned versions once src/emma/ lands.)

  • Command surface precedent. The one-command-per-capability shape mirrors MLPerf’s mlperf_* submission and scoring commands (Mattson et al., 2020), where the CLI is the auditable interface to the scoring path. SUPERB’s superb runner is the precedent for a single entry point driving a frozen-backbone, many-readouts pipeline (Yang et al., 2021). (verify)

  • Structured output. Every command emits columnar text or JSON so it pipes into scripts, following the Unix-tool convention and the JSON-lines convention used by modern ML harnesses.

Library reuse is checked: no CLI argument parsing, hashing, or schema validation is reimplemented; each delegates to Typer, the DataRelease, and SubmissionValidator respectively.

5. Experimental methodology and planned tests

The CLI has no hyperparameters to sweep; its tests are contract tests, not experiments. Three suites, proposed so an implementer can drop them into tests/validation/cli/. Until src/emma/ lands, each test is gated to skip-with-rationale rather than collect-as-failure; the active lens emits the plan plus the literature evidence, not a green check.

Suite CLI-T1: signature introspection (construct). Walk the Typer application’s registered commands. Assert the command names, subcommand set, flag names, flag types, defaults, and required-ness equal the table in API / CLI. The frozen signature table is the contract; a drift between docs and code fails the build.

# tests/validation/cli/test_signatures.py (proposed)
def test_command_set_matches_api_reference(typer_app):
    commands = {c.name for c in typer_app.registered_commands}
    assert commands == {"list-tasks", "download", "eval", "submit", "score", "release"}

def test_download_rejects_holdout(typer_app):
    # --split holdout must be a rejected value, not a silent path.
    assert "holdout" not in accepted_split_values(typer_app, "download")

def test_protocol_flag_uses_snake_case_enum(typer_app):
    # --protocol accepts only snake_case OODProtocol values, not kebab-case aliases.
    accepted = accepted_protocol_values(typer_app, "eval")
    assert accepted == {"leave_one_environment_out", "leave_one_unit_out", "leave_one_band_out"}
    assert "leave-one-environment-out" not in accepted

Suite CLI-T2: default-behavior (construct). Assert that emma eval with no --protocol uses the task’s OODAxis and returns a per-fold mean; that emma download resolves a DatasetRecipe and produces a content hash matching a re-derivation; and that emma list-tasks --format json round-trips through the TaskRegistry with each task’s reported metric matching its TaskSpec (in particular E-ID-DRONE reports AUC; F1 is scoped to E-S2T-QA and is absent from the v0.1 listing).

Suite CLI-T3: output fidelity. Assert the console values printed by emma eval equal the EvalRun fields to floating-point equality, and that emma eval writes a RunManifest with every documented field populated.

Sample sizes are one invocation per assertion; the tests are deterministic, so statistical power does not apply. The gold-standard reference is the API / CLI signature table itself.

6. Robustness boundaries

Where the CLI must fail, and how. Each row maps an invalid input to the error the harness raises; the test asserts both the failure and the error type.

Input

Failure mode

Error

Unknown or unversioned task id, for example --task E-NO-SUCH

Reject before any work runs

TaskNotReleased

emma download --split holdout

Reject; the holdout never exists outside the scoring path

SplitNotAvailable

Recipe references an rfgen commit or config not installed

Reject with the missing pin named

RecipeMismatchError

Prediction Parquet missing scene_id, wrong columns, or wrong count

Reject locally; no upload attempted

PredictionSchemaError

emma eval --split holdout outside the scoring server

Reject; holdout access is gated

HoldoutAccessError

The operating envelope: the CLI assumes rfgen is installed and resolvable for any download or release command, and assumes network access only for submit. Outside that envelope the commands fail with a named error, never a silent wrong answer.

Proposed sweep: for each command, fuzz the enum-typed flags (--split, --version, --task, --protocol) with values outside the closed set, including kebab-case aliases such as leave-one-environment-out that the snake_case OODProtocol value set rejects, and assert a rejection. Closed-set enforcement is the contract (see Principles, encode rules).

7. Synthesis

The CLI is a thin, fully enumerable surface, so its validation is contract testing rather than empirical research. The load-bearing claims are the four construct claims (CLI-C1 through CLI-C4), and the primary guard is the signature-introspection suite (CLI-T1), which fails the build on any drift between the Typer app and the documented command table.

Top three findings.

  1. (high) The download-regenerates claim (CLI-C2) is the sharpest construct risk. The command name and behavior diverge on purpose; a static-blob regression would break reproducibility silently. The regeneration-and-hash test is mandatory before v0.1 ships.

  2. (high) The default-OOD behavior (CLI-C3) must be asserted, not assumed. A default that silently fell back to in-distribution accuracy would invalidate the leaderboard. The default-behavior suite (CLI-T2) is mandatory.

  3. (medium) Closed-set flag enforcement is the robustness contract. Invalid task ids, missing recipes, and schema-mismatched predictions must all fail with a named error before any work or upload runs; the robustness suite asserts every row of the table in section 6.

No architectural deferrals. The two (verify) library citations (Typer pinned version; SUPERB/MLPerf command-surface precedent) resolve once src/emma/ lands and the dependency manifest is pinned.

References

  • Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. The auditable CLI-as-scoring-interface precedent and the one-command-per-capability shape. (verify section relevance)

  • Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. The single-entry-point runner for a frozen-backbone, many-readouts pipeline. (verify)

  • Typer, “Typer, build great CLIs.” The CLI framework the harness adopts; to be pinned once src/emma/ lands. (verify version)

See Also