Primitives validation

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

Scope: the construct, literature, and methodology validation of the emma.enums closed sets and the emma.errors exception hierarchy documented in Primitives. As an engineering layer, this report carries construct, literature, and methodology depth; the mathematical-fidelity, empirical-realism, and robustness-boundaries lenses are lighter here because primitives carry no formulas and no data. The contract under validation is that the closed sets are correct and complete, and that each failure mode raises the exception that names it.

1. Purpose and construct

The primitives layer is the vocabulary every later layer reads. A TaskID member that does not exist cannot be scored; a Split member that leaks into the wrong column breaks the held-out contract. Construct validity asks two questions: are the closed sets correct and complete, and does each error class name the failure mode it guards.

Closed-set completeness

Each enumeration closes a set that the rest of the benchmark treats as exhaustive. The construct check is that the declared members cover every value the reference and API pages use, with no extras and no gaps.

• TaskID declares 15 members: three localization, five identity, two channel, three scene understanding, and two signal-to-text. The count matches the 15 tasks in Task reference and the 15 datasets in Datasets. A 16th task proposed without a member is unscorable by construction; that is the desired gate.

• OODAxis declares 8 dimension-style members (ENVIRONMENT, DEVICE, FREQUENCY_BAND, SNR_REGIME, DRONE_MODEL, WAVEFORM_FAMILY, EMITTER_MIX, SCENE_TYPE); the LEAVE_ONE_..._OUT names that appear in Task reference are OODProtocol fold-construction names, not axis members. The construct check is that every task’s primary and secondary axis in Task reference resolves to a member, and that no task names a free-form axis string. Six non-prose primaries and three secondaries already resolve to a member (E_ID_RWAVE to WAVEFORM_FAMILY, E_ID_AMC to SNR_REGIME, E_CH_BEAM / E_S2T_CAP / E_S2T_QA to SCENE_TYPE, E_SC_SEP to EMITTER_MIX; secondaries E_LOC_POS to DEVICE, E_ID_DRONE to DRONE_MODEL, E_LOC_LOS to FREQUENCY_BAND). Four tasks (E_ID_UAVDOP, E_CH_CSI, E_SC_ANOM, E_SC_SENSE) still carry prose axes (“unseen platform + motion profile”, “unseen scenario”, “unseen normal baseline”, “unseen band + noise floor”) that await formalization; this is an open consistency item, not a missing member.

• Domain declares 6 members covering the communities each task and recipe declares.

• Split declares 3 members; the construct rule is that Split.HOLDOUT labels never appear in a public artifact.

• Environment declares 3 members; two synthetic (E_URBAN, E_RURAL) make leave-one-environment-out live at v0.1, and E_REAL_CAP is fidelity only.

• Pillar (5), ReleaseVersion (4), Maturity (5), SubmissionMode (2), and MetricDirection (2) round out the closed choices.

Error-class fit

Each error class names one failure mode so callers can discriminate handling. The construct check is that the raised class matches the guard that fired.

• TaskNotReleased: TaskRegistry is asked for a task outside its ReleaseVersion. Correct: the failure is version gating, not a missing task.

• SplitNotAvailable: a caller requests a split it cannot access, the canonical case being holdout labels outside the scoring path. Correct.

• RecipeMismatchError: DataRelease resolves scenes whose content hash does not match the DatasetRecipe pin. Correct: the failure is provenance drift.

• PredictionSchemaError: SubmissionValidator rejects a malformed PredictionBundle. Correct.

• SubmissionRejected: a submission fails validation after the bundle schema passes, for example an AntiOverfit rate-limit or a missing PreRegistration. Correct: the failure is leaderboard policy, not schema.

• HoldoutAccessError: an unauthorized read of holdout labels or execution outside the sandbox. Correctly distinct from SplitNotAvailable: the former is a governance boundary violation recorded in the audit log, the latter is the general access case.

Sharpest construct threats

(Severity: medium) Four tasks in Task reference express their OOD (out-of-distribution) axis in prose rather than as a pinned OODAxis member (E_ID_UAVDOP, E_CH_CSI, E_SC_ANOM, E_SC_SENSE). Until those map to constants, the closed set is enforced by convention, not by the enum. Open: pin each task’s primary axis to a member constant on the TaskSpec.

(Severity: low) Cross-page consistency between the canonical OODAxis contract and the per-task wiring in Task reference. The OODAxis table in Primitives is the canonical source and correctly enumerates the 8 dimension-style members (ENVIRONMENT, DEVICE, FREQUENCY_BAND, SNR_REGIME, DRONE_MODEL, WAVEFORM_FAMILY, EMITTER_MIX, SCENE_TYPE). The v0.1 rows in Task reference write the OOD axis column with OODProtocol fold-construction names (LEAVE_ONE_ENVIRONMENT_OUT, LEAVE_ONE_BAND_OUT, LEAVE_ONE_UNIT_OUT), which are protocol names rather than axis members, while the later-task rows use dimension-style members (WAVEFORM_FAMILY, SNR_REGIME, SCENE_TYPE, EMITTER_MIX). Rewriting the v0.1 axis column to name the underlying axis members (ENVIRONMENT, FREQUENCY_BAND, DEVICE) is a doc-consistency open item, not a contract gap: the canonical enum is correct, the reference column is the drift.

(Severity: low) The context: dict[str, object] attribute on EmmaError is typed object but documented as JSON-serializable. The construct depends on every raiser honoring JSON-serializability; a non-serializable value breaks telemetry silently. Open: constrain the type or assert serializability in the base class.

2. Mathematical fidelity

Primitives carry no formulas. The only arithmetic in the layer is the enum member count, audited in section 1: TaskID has 15 members, OODAxis has 8, and the per-pillar and per-domain distributions match Task reference. No metric or aggregation math lives here.

3. Empirical realism

Primitives have no data surface, so empirical realism is not a load-bearing lens for this layer. The relevant realism property is observational: every closed set was populated from the tasks, datasets, and governance decisions recorded elsewhere in the spec, and the counts match those sources. Section 1 records that observation.

4. Literature grounding

The primitives layer reuses two established mechanisms rather than inventing its own.

• StrEnum for closed choices. The closed-set convention reuses Python’s enum.StrEnum so that YAML and recipe configs use the string value while Python code uses the member. This follows the STYLE.md enum-versus-string rule and matches the rfgen core-type convention. No hand-rolled string-registry is introduced for closed choices.

• Pydantic v2 coercion. Enum values in serialized configs deserialize through Pydantic v2 (pydantic >= 2.0), so a YAML string becomes the member at validation time. The library, not EMMA, owns the coercion.

• Open plugin registries stay str. Names that resolve through a registry (a custom RealCaptureAdapter name) remain plain strings because they are open, not closed. This is the documented escape valve and prevents the closed-set gate from blocking extension.

The exception hierarchy reuses the standard Python Exception base with a single root (EmmaError) so callers can catch all benchmark faults with one except clause while discriminating on subclasses. No novel error mechanism is introduced.

(Severity: low) No new literature entries are required for this report; every mechanism above is a standard-library or established-library reuse.

5. Experimental methodology and planned tests

Tests are written so an implementer can port them directly to tests/validation/primitives/ once src/emma/ lands. The primitives layer is amenable to exhaustive contract tests because the sets are small and closed.

Enum-completeness contract - tests/validation/primitives/test_enum_completeness.py

• Falsifiable claim. Every closed set is exhaustive against the rest of the spec: every TaskID has a TaskSpec, every task’s metric id resolves to a Metric subclass, and every task’s OOD axis resolves to an OODAxis member.

• Failure mode. A task is declared in TaskID but has no spec, or a spec’s metric_id names a class that does not exist, or a task’s axis is a free-form string.

• Design. Iterate the TaskID members; assert each resolves through TaskRegistry at its ReleaseVersion; assert each spec’s metric_id imports to a concrete metric class; assert each spec’s ood_axis is an OODAxis member, not a bare string.

• Sample size. All 15 members (exhaustive).

• Statistical test. Assertion-based; a missing mapping fails the build.

• Tolerance. Zero tolerance; every member must resolve.

• Gold-standard reference. The task table in Task reference and the metric table in Metrics reference.

Member-count regression - tests/validation/primitives/test_member_counts.py

• Falsifiable claim. The declared member counts match the spec: 15 tasks, 8 OOD axes, 6 domains, 3 splits, 3 environments.

• Failure mode. A member is added or removed without a matching spec change, silently shifting the closed set.

• Design. Assert len(list(TaskID)) == 15, and the corresponding counts for the other enums; fail on drift.

• Statistical test. Exact equality.

• Tolerance. Zero.

• Gold-standard reference. The class-index table in Primitives.

Error-routing contract - tests/validation/primitives/test_error_routing.py

• Falsifiable claim. Each guard raises the error class that names its failure mode, and every benchmark error is an EmmaError subclass.

• Failure mode. A holdout read raises a generic Exception instead of HoldoutAccessError, or a schema failure raises SubmissionRejected instead of PredictionSchemaError.

• Design. Trigger each failure path and assert the raised class; assert isinstance(raised, EmmaError) for every subclass.

• Sample size. One trigger per error class (7 paths).

• Statistical test. Assertion-based.

• Tolerance. Zero tolerance on class identity.

• Gold-standard reference. The exception hierarchy in Primitives.

6. Robustness boundaries

• Prose axes outside the enum. (Severity: medium) Until the four prose OOD axes are pinned to constants, the closed set is partly conventional. Probe: the enum-completeness test in section 5 fails if a spec carries a bare string axis.

• Open-set leakage. (Severity: low) A contributor who adds a task using a Literal[...] instead of a TaskID member bypasses the gate. Probe: a grep-level check that no Literal over task identifiers appears in src/emma/; the smell test is documented in PRINCIPLES.md.

• Non-serializable context. (Severity: low) Covered in section 1; a raiser that attaches a non-JSON-serializable object to context breaks telemetry. Probe: assert json.dumps(err.context) succeeds in the error-routing test.

7. Synthesis

Load-bearing claims supported. The closed sets are exhaustive against the rest of the spec at the declared counts (15 tasks, 8 OOD axes, 6 domains, 3 splits, 3 environments), and each error class names the failure mode its guard detects. The StrEnum plus Pydantic coercion model reuses established mechanisms without inventing a registry.

Gaps surfaced. (1) Four tasks express their OOD axis in prose rather than as a pinned member, leaving part of the closed set conventional. (2) The v0.1 rows of the OOD axis column in Task reference use OODProtocol fold-construction names (LEAVE_ONE_..._OUT) rather than the underlying OODAxis members; the canonical OODAxis table in Primitives is correct. (3) The context attribute is typed object but must be JSON-serializable, a contract enforced by convention until a base-class assertion lands.

Recommended actions. (1) Pin every task’s primary axis to an OODAxis member on its TaskSpec. (2) Rewrite the v0.1 rows of the OOD axis column in Task reference to name the underlying OODAxis members (ENVIRONMENT, FREQUENCY_BAND, DEVICE) rather than OODProtocol names; the canonical OODAxis table in Primitives is already correct. (3) Land the enum-completeness and error-routing contract tests as the first tests/validation/primitives/ entries. (4) Assert JSON-serializability of context in the EmmaError base class.

Top three findings.

1. (medium) Four tasks carry prose OOD axes that are not pinned to OODAxis constants; the closed set is partly conventional until the specs pin them.

2. (low) The v0.1 rows of the OOD axis column in Task reference use OODProtocol fold-construction names rather than the underlying OODAxis members; the canonical OODAxis table in Primitives is correct and the reference column is the drift.

3. (low) The enum-completeness contract test (every TaskID resolves to a TaskSpec and a metric) is the proposed first verification and is straightforward to port.

4. (low) The error hierarchy is well-fit to its failure modes; the error-routing contract test closes the residual risk of a generic raise.

References

• Python enum.StrEnum, standard library (Python 3.11+; enum.StrEnum). The closed-set convention every EMMA enum reuses so YAML and recipe configs deserialize by string value while Python code uses the member.

• pydantic >= 2.0. Enum coercion at validation time, ValidationError emission for out-of-set values, and required-field enforcement.

• Python Exception hierarchy. The single-root pattern (EmmaError) lets callers catch all benchmark faults with one except clause while discriminating on subclasses.

No external scientific claims are made on this page; the load-bearing citations are standard-library and established-library reuse.

See Also

• Primitives: the enum and error contracts this report validates.

• Task reference: the per-task axes and metrics the completeness test checks against.

• Schemas validation: the Pydantic models that consume these enums.

• Validation methodology: the seven-section framework and the engineering-layer posture.

• Literature: the master reading list (no new entries required for this report).