Schemas validation

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

Scope: the construct, literature, and methodology validation of the Pydantic v2 models and frozen dataclasses documented in Schemas and the Schema reference pages. As an engineering layer, this report carries construct, literature, and methodology depth; the empirical-realism and robustness-boundaries lenses are lighter because schemas carry no data and no formulas. The contract under validation is that every model enforces its contract at parse time, and that every record round-trips losslessly through its serialized form.

1. Purpose and construct

A schema is the parse-time contract. A Submission that parses has already passed the structural checks a reviewer would otherwise do by hand: required fields present, closed-set enums coerced, hashes in the expected format. Construct validity asks two questions: do the schemas enforce the contracts at parse time, and do the records round-trip losslessly.

Parse-time enforcement

  • Required fields. Every model marks the fields a contract depends on as required; optional fields carry an explicit default. TaskSpec declares no optional fields, so a spec missing its OOD (out-of-distribution) axis fails to parse rather than silently scoring under no axis.

  • Closed-set enums via coercion. Fields typed as an enum (TaskID, Split, ReleaseVersion, Domain, Maturity) deserialize a YAML string value into the member at validation time. A value outside the closed set raises a Pydantic ValidationError, not a silent fallback. This is the parse-time half of the enum gate audited in Primitives validation.

  • Hash formats. Hash-bearing fields record the algorithm prefix (sha256:) so a malformed hash is detectable: content_hash on SceneManifest and DataReleaseManifest, prediction_bundle_hash on PredictionBundle, the per-task prediction_bundle_hashes dict on RunManifest, data_release_hash on LeaderboardRow and RunManifest, and the PreRegistration hash. (Severity: low) Open: pin a regex or constrained type so a hash without the prefix fails validation, not just inspection.

Round-trip losslessness

The record models that anchor auditability must survive a serialize, deserialize, and re-serialize cycle unchanged.

  • PredictionBundle pins which outputs were scored via prediction_bundle_hash; the rows path points to the Parquet file the hash was computed over.

  • Submission carries one or more bundles plus the model_provenance and optional pre_registration_id that a LeaderboardRow inherits.

  • LeaderboardRow records two halves of the auditable result triple on the public board: data (data_release_hash) and metric (task_scores plus ood_avg, with sim_real_gap as the fidelity column). It deliberately does not carry the per-task prediction-bundle hashes; that model half lives on the linked run record so the row stays small.

  • RunManifest carries the model half the row omits, the per-task prediction_bundle_hashes dict, alongside data_release_hash, task_metrics, and the seed and harness_version that make the readout-head fit reproducible. The run manifest is the local-evaluation counterpart to the row, written for any run whether posted or not.

The construct claim is that parsing a serialized record and re-serializing it yields byte-identical canonical output, so a reviewer comparing a posted hash to a re-derived record sees no drift. The row/run hash split is the load-bearing separation: a reviewer recovers the model half of the triple from the linked RunManifest, not from the row.

Helper types

Three helper types in Schemas carry no parse-time validation beyond their declared shape, so their construct claims sit one layer down.

  • SeedRange (a frozen dataclass) closes an integer seed interval [start, stop] and declares the realized scene count. The construct rule is that the three split ranges partition the seed space without overlap and that count agrees with the interval length when the range is contiguous (stop - start + 1). Enforcement is loader-level, not parse-time, because SeedRange is a frozen dataclass.

  • MetadataPath (a str alias) names a path into realized rfgen metadata via the grammar <container>[].<rfgen_key> (canonical case: emitters[].aoa_deg for the AoA, angle-of-arrival, regression label). The grammar is a documented constraint on a plain string alias, so a free-form path parses today and fails at load time when LabelExtractor cannot resolve it.

  • EmitterRecord (a frozen dataclass) promotes a subset of the rfgen per-emitter metadata bag (aoa_deg, los_flag) onto SceneManifest. The raw bag travels separately; the record is the contract surface a recipe’s label_extraction targets.

The construct claim is that the helper types are EMMA-owned contracts over rfgen metadata and split bookkeeping, not rfgen types; the fields they expose are the surface a recipe may target, and the fields they omit stay in the raw bag.

Sharpest construct threats

(Severity: medium) The frozen dataclasses (TaskSpec, PredictionBundle) are declared frozen=True, slots=True but are not Pydantic models, so they do not inherit Pydantic validation. A TaskSpec constructed in Python with a bare string ood_axis instead of an OODAxis member bypasses the coercion that a YAML-loaded DatasetRecipe gets. Open: either route construction through a validating factory, or document that in-Python construction is trusted code.

(Severity: low) The DatasetRecipe label_extraction map values are typed MetadataPath (a str alias). A path that does not exist in the realized rfgen metadata yields an empty label silently at load time, not at parse time. This is covered in Datasets validation; the schema-side mitigation is a loader-level check, not a parse-time field.

2. Mathematical fidelity

Schemas carry no metric formulas. Two arithmetic invariants live at this layer.

  • Hash composition. The DataReleaseManifest content_hash is computed over the manifest plus its referenced SceneManifest hashes, and the same data_release_hash value propagates onto LeaderboardRow and RunManifest. The composition must be associative and order-stable so re-serializing a release does not change its hash. (Severity: low) Covered in Content hashing; the schema-side contract is that the manifest serializes to canonical JSON before hashing.

  • SeedRange interval arithmetic. Each split maps to a closed integer interval [start, stop] with a declared count. The invariant is count == stop - start + 1 for a contiguous range, and the three split intervals are pairwise disjoint so a scene cannot belong to two splits. (Severity: low) The arithmetic is enforced at the loader level, not at parse time, because SeedRange is a frozen dataclass; the schema-side contract is that the three intervals are well-formed before the loader sees them.

3. Empirical realism

Schemas have no data surface, so empirical realism is not a load-bearing lens here. The relevant property is observational: every field documented on the Schema reference pages matches the field list on the Schemas API page, including the helper types (SeedRange, MetadataPath, EmitterRecord) and the prediction_bundle_hashes dict on RunManifest. The two pages were checked to agree during this report; the row/run hash split (data-release hash on the row, per-task prediction-bundle hashes on the run) is reflected on both.

4. Literature grounding

The schemas layer reuses established mechanisms rather than inventing its own.

  • Pydantic v2 for validation. Every manifest and record model is a Pydantic v2 BaseModel (pydantic >= 2.0), so required-field enforcement, enum coercion, and ValidationError emission are library behavior, not hand-rolled checks. This follows the PRINCIPLES.md encode-don’t-describe rule.

  • Frozen dataclasses for data types. TaskSpec and PredictionBundle are dataclass(frozen=True, slots=True), matching the rfgen core-type convention. The choice keeps immutable data types separate from validated manifest models, the three-roles separation documented in API reference.

  • Helper types follow the same convention. SeedRange and EmitterRecord are dataclass(frozen=True, slots=True) matching the data-type role above; MetadataPath is a constrained str alias whose grammar (<container>[].<rfgen_key>) is documented rather than type-enforced. The split keeps the helper types out of the validated-model set while still pinning the surface a recipe targets.

  • Canonical serialization. Hash-bearing models serialize to canonical JSON (stable key order, no comments, no whitespace variation) so the content hash is a pure function of the resolved content. See Content hashing.

(Severity: low) No new literature entries are required for this report; every mechanism is a library reuse. The Parquet format used for prediction rows is cited on the Prediction bundle page and listed in Literature.

5. Experimental methodology and planned tests

Tests are written so an implementer can port them directly to tests/validation/schemas/ once src/emma/ lands. The schemas layer is amenable to table-driven contract tests because the field sets are finite and the round-trip is mechanical.

Round-trip losslessness - tests/validation/schemas/test_round_trip.py

  • Falsifiable claim. Serializing a record, deserializing the result, and re-serializing yields byte-identical canonical output for every record model.

  • Failure mode. A field loses precision, an enum round-trips to a different string value, or a dict key order changes the hash.

  • Design. For each of DatasetRecipe, SceneManifest, DataReleaseManifest, Submission, LeaderboardRow, RunManifest, PreRegistration, and ReScoringLog, construct a representative instance, serialize to canonical JSON, deserialize, re-serialize, and assert the two serializations are identical.

  • Sample size. One instance per model, plus one edge instance per model with optional fields populated and empty.

  • Statistical test. Exact byte equality of the canonical serialization.

  • Tolerance. Zero.

  • Gold-standard reference. The field tables on the Schema reference pages.

Validation-rejection suite - tests/validation/schemas/test_validation_rejects.py

  • Falsifiable claim. A record that violates its contract fails to parse with a specific, named error rather than silently coercing.

  • Failure mode. A Submission missing its prediction_bundles parses to an empty list, or a DatasetRecipe with a non-member task_id coerces to a string.

  • Design. Feed each model a malformed payload: missing required field, out-of-enum value, wrong-type value, malformed hash prefix. Assert Pydantic raises ValidationError and that the error locates the offending field.

  • Sample size. One malformed payload per failure class per model.

  • Statistical test. Assertion-based; the raise must occur and must name the field.

  • Tolerance. Zero tolerance; a malformed record must never parse.

  • Gold-standard reference. The required-field and enum tables on the Schemas page.

Hash-composition stability - tests/validation/schemas/test_hash_composition.py

  • Falsifiable claim. The DataReleaseManifest hash is stable under re-serialization and changes when any referenced scene hash changes.

  • Failure mode. Re-serializing a release changes its hash, or a tampered scene hash leaves the release hash unchanged.

  • Design. Build a release manifest, hash it, re-serialize, hash again, assert equality; mutate one scene hash, re-hash, assert inequality.

  • Sample size. One release with at least two scenes.

  • Statistical test. Exact equality on re-serialization; exact inequality on tampering.

  • Tolerance. Zero.

  • Gold-standard reference. The content-hash construction in Content hashing.

SeedRange invariants - tests/validation/schemas/test_seed_range.py

  • Falsifiable claim. The three split SeedRange intervals in a recipe are pairwise disjoint, and for each range count == stop - start + 1 when the range is contiguous.

  • Failure mode. Two split intervals share a seed (a scene could be scored under two splits), or a declared count disagrees with the interval length.

  • Design. Parse a DatasetRecipe, collect the SeedRange for each Split member, assert pairwise-empty intersection of the closed [start, stop] intervals, and assert count == stop - start + 1 for each contiguous range. Holdout seeds scanned against public manifests is covered in Datasets validation.

  • Sample size. One recipe per released dataset (exhaustive over v0.1).

  • Statistical test. Exact set-equality on interval intersection; exact equality on count.

  • Tolerance. Zero overlap; exact count agreement.

  • Gold-standard reference. The SeedRange helper-type contract in Schemas.

MetadataPath resolution - tests/validation/schemas/test_metadata_path.py

  • Falsifiable claim. Every MetadataPath in a recipe’s label_extraction map matches the <container>[].<rfgen_key> grammar, and a path whose key rfgen renamed surfaces as a loader-level RecipeMismatchError rather than an empty split.

  • Failure mode. A free-form path (for example emitters.aoa_deg without the [] token) parses silently, or a path whose key rfgen renamed yields an empty split.

  • Design. Assert each label_extraction value matches the grammar; assert each path resolves on a sample realized scene. The split-wide miss case is covered in Datasets validation.

  • Sample size. Every label_extraction entry across v0.1 recipes (the canonical case is emitters[].aoa_deg).

  • Statistical test. Regex match on the grammar; assertion that the resolved key exists in the rfgen metadata bag for a sample scene.

  • Tolerance. Zero; a path that misses the grammar or the metadata fails.

  • Gold-standard reference. The MetadataPath grammar and the EmitterRecord promoted fields (aoa_deg, los_flag) in Schemas.

RunManifest hash presence - tests/validation/schemas/test_run_manifest_hashes.py

  • Falsifiable claim. A RunManifest carries a prediction_bundle_hashes entry for every TaskID in its task_metrics, and each entry carries the sha256: prefix.

  • Failure mode. A run records a metric for a task but omits its bundle hash, breaking the model half of the audit triple, or a hash loses its prefix and becomes uninspectable.

  • Design. Construct a run manifest with a representative task_metrics dict; assert prediction_bundle_hashes has the same key set; assert each value matches the hash regex. The row-side counterpart, the LeaderboardRow that does not carry bundle hashes, is asserted as a negative: constructing a row with a bundle-hash field raises.

  • Sample size. One manifest covering the v0.1 scored tasks; one negative row instance.

  • Statistical test. Exact set-equality on keys; regex match on hash values.

  • Tolerance. Zero; a missing hash or a missing prefix fails.

  • Gold-standard reference. The RunManifest and LeaderboardRow field tables in Schemas.

6. Robustness boundaries

  • Trusted in-Python construction. (Severity: medium) Covered in section 1; frozen dataclasses do not validate on construction. Probe: a factory that validates enum fields on TaskSpec creation, or a ban on direct construction outside the registry.

  • Label-path silence. (Severity: medium) A label_extraction path that misses the realized metadata yields an empty label at load time, not a parse error. Probe: the loader-level label-path resolution test in Datasets validation.

  • MetadataPath grammar escape. (Severity: low) MetadataPath is a str alias constrained by documented grammar, not by a parse-time type. A free-form string parses today and fails downstream. Probe: the MetadataPath resolution test in section 5; a constrained type or regex would make the grammar parse-time.

  • SeedRange overlap. (Severity: low) A recipe whose split intervals overlap is not caught at parse time because SeedRange is a frozen dataclass. Probe: the SeedRange invariants test in section 5.

  • RunManifest hash omission. (Severity: low) A run that records a metric for a task but omits the matching prediction_bundle_hashes entry breaks the model half of the audit triple silently. Probe: the RunManifest hash presence test in section 5.

  • Hash-prefix drift. (Severity: low) A hash field without the sha256: prefix is not caught at parse time today. Probe: a constrained string type or regex on hash-bearing fields, exercised by the validation-rejection suite.

7. Synthesis

Load-bearing claims supported. The Pydantic v2 models enforce their contracts at parse time (required fields, closed-set enum coercion, hash formats by convention), the helper types (SeedRange, MetadataPath, EmitterRecord) pin the surface a recipe targets, and the row/run hash split keeps the auditable result triple recoverable: the row carries the data and metric halves, the run carries the per-task model half. Library reuse (Pydantic, frozen dataclasses, canonical JSON) carries the validation rather than hand-rolled checks.

Gaps surfaced. (1) The frozen dataclasses, including the helper types, do not validate on in-Python construction, so trusted code can bypass the enum coercion that YAML-loaded records get. (2) Hash fields carry the algorithm prefix by convention but not by a constrained type. (3) Label-path misses and split overlaps are silent at parse time and require loader-level checks. (4) The MetadataPath grammar is documented but not type-enforced.

Recommended actions. (1) Land the round-trip, validation-rejection, SeedRange, MetadataPath, and RunManifest hash-presence suites as the first tests/validation/schemas/ entries. (2) Route TaskSpec construction through a validating factory, or document the trusted-code boundary. (3) Constrain hash-bearing fields with a regex or constrained type so a missing prefix fails at parse time. (4) Constrain MetadataPath with a regex or constrained type so a free-form path fails at parse time.

Top three findings.

  1. (medium) Frozen dataclasses (TaskSpec, PredictionBundle, SeedRange, EmitterRecord) skip Pydantic validation on construction; a factory or a documented trusted-code boundary closes the gap.

  2. (low) The row/run hash split is the load-bearing audit separation: the row carries the data-release hash and the metric columns, the run carries the per-task prediction-bundle hashes; the RunManifest hash-presence test pins it.

  3. (low) The round-trip losslessness and validation-rejection suites are the proposed first verification and are straightforward to port.

  4. (low) Hash-bearing fields and MetadataPath enforce their formats by convention; constrained types would make both checks parse-time.

References

  • pydantic >= 2.0. Required-field enforcement, closed-set enum coercion, constrained types, and ValidationError emission for the manifest and record models.

  • Python dataclasses (PEP 557) with frozen=True, slots=True. The immutable data-type pattern for TaskSpec and PredictionBundle, matching the rfgen core-type convention.

  • Apache Parquet format, referenced on Prediction bundle. The columnar storage format for prediction rows.

No external scientific claims are made on this page; the load-bearing citations are library reuse.

See Also