Harness validation

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

Scope: the six-lens validation of the evaluation harness, concentrated on the Evaluator entry point, the ProtocolRunner fold-construction driver, and the leave-one-environment-out protocol that scores the four v0.1 tasks. The per-metric formula audit lives in Metrics validation; the per-task construct lives in Tasks validation. This report asks whether the harness that wires them together actually measures transfer.

1. Purpose and construct

EMMA (Electromagnetic Multi-task Model Assessment) scores one frozen backbone on one task under its OOD (out-of-distribution) protocol, via Evaluator. The construct claim is that the resulting score measures transfer, not in-distribution accuracy: every contribution to the reported mean comes from a held-out value the model never trained on. Construct validity asks whether the harness realizes that property, and whether the readout-head re-fit per fold is the right isolation.

  • Claim. Under leave-one-environment-out, a model that learned transferable features scores above the per-task chance baseline on the held-out environment; a model that overfit the training environment’s channel statistics does not.

  • Proxy. ProtocolRunner materializes one fold per held-out environment, fits the readout head on the training fold, scores the eval fold with the task metric, and reduces the fold scores. The frozen backbone is shared across folds; only the lightweight readout head is re-fit. The fold-reduction is OODAvg, the sole Aggregator subclass at v0.1; SimToRealGap is a standalone fidelity-gap reporter, not an Aggregator subclass, and produces the gap column alongside the OOD average, never inside it. The output is an EvalRun carrying per-environment metrics and the aggregated OOD average, plus the RunManifest the harness emits to pin what ran.

  • Omissions. The harness scores one task at a time; cross-task transfer is an aggregate property of the board, not of a single run. The readout head’s capacity is not itself audited for being “lightweight enough”.

  • Downstream risk. If the train or eval fold leaks across the protocol boundary, the score stops measuring transfer and measures memorization instead. The fold boundary is the construct; a mis-constructed fold invalidates the run silently. The harness-emitted RunManifest records what ran, not whether the fold was correct: its per-task prediction_bundle_hashes pin exactly which outputs were scored, so a reviewer can re-load the scored rows, but a leaked scene still posts a high score unless a guard rejects it.

  • Domain utility. The harness produces the transfer scores that positioning, defense, spectrum, and communications communities consume as the benchmark result. A harness that measures in-distribution accuracy would re-import the failure mode EMMA exists to move beyond.

Sharpest construct threats

(Severity: high, architectural) The fold boundary is the entire construct. ProtocolRunner must guarantee that every scene from the held-out environment goes to the eval fold and no scene from it leaks to the training fold. A single mis-assigned scene is a silent leak that inflates the score without any visible error. The mitigation is a guard test (section 5) that swaps the train and eval split and checks the score collapses, rather than relying on code inspection.

(Severity: medium) The readout head is re-fit per fold, which is correct, but its capacity is a free parameter. A head with enough capacity can memorize the training fold and still fail to transfer; a head with too little capacity underfits. SUPERB (Speech processing Universal PERformance Benchmark) pins lightweight readouts to keep the score a reflection of representation quality; EMMA inherits that contract, and the head capacity must be pinned per task, not tuned per submission.

2. Mathematical fidelity

The harness owns no metric formula of its own; it composes the per-task metric (audited in Metrics validation) with the fold-reduction mean. Two harness-side math checks apply.

  • Fold-reduction mean. The reported score is the arithmetic mean of per-environment metric values, not a scene-weighted mean. Weighting by scene count would let a high-scoring environment with many scenes dominate; the unweighted mean treats each held-out environment as one observation, which is the unit the protocol holds out. (Severity: low) Open: confirm ProtocolRunner uses the unweighted environment mean, matching the formula in OOD protocol.

  • Aggregation normalization. OODAvg, the sole Aggregator subclass at v0.1, normalizes each task score with frozen anchors before averaging. The harness must pass the raw per-environment mean to OODAvg without pre-normalizing, or the frozen anchors are bypassed. SimToRealGap is never on this code path: its pairwise (synth_score, real_score) signature is not substitutable for the __call__(fold_scores) contract, so subclassing Aggregator would be a Liskov violation. (Severity: medium) Open: confirm the raw-to-normalized handoff is single-sourced in OODAvg, not duplicated in the runner.

  • Seed handling. RunManifest records one seed per run that pins both fold construction and readout-head fitting. (Severity: medium) Open: confirm the seed is threaded end-to-end through the fold sampler and the readout-head fitter, not just recorded after the fact, so a re-run with the same seed reproduces the run bit-for-bit (the seed-determinism test in section 5). The prediction_bundle_hashes field then pins the per-task outputs that came out of that seeded fit.

3. Empirical realism

The harness consumes the pinned rfgen (radio-frequency signal generation) recipes validated in Datasets validation. The harness-side realism question is whether the leave-one-environment-out fold it constructs is physically meaningful: do the held-out environments (E_URBAN, E_RURAL) differ enough in channel statistics that a model cannot trivially transfer between them?

  • The v0.1 recipe ships at least two synthetic environments. Whether two is enough to make the held-out environment a genuine transfer test is the load-bearing realism question for the harness, and it is underpowered (section 5).

  • Real-capture OOD subsets (Colosseum, POWDER, COSMOS) are not wired at v0.1, so the harness’s sim-to-real path is asserted by design, not yet exercised. The SimToRealGap reporter (standalone, not an Aggregator subclass) runs once the RealCaptureAdapter set lands; it consumes the synthetic OOD average and the real-capture score as a pair and emits the gap as a separate column alongside the aggregate, never folded into it.

(Severity: medium) The two-environment design is realistic in kind (two distinct channel environments do differ) but not in statistical power. The realism and the power concerns are the same concern viewed two ways: two environments is a real transfer test for a single comparison, but a fragile basis for an aggregate.

4. Literature grounding

The canonical sources, drawn from the master reading list:

  • Leave-one-X-out. Koh et al. (WILDS) is the distribution-shift-first precedent; the protocol the runner implements is the WILDS leave-one-X-out framing applied to the environment axis.

  • One backbone, lightweight readouts, one aggregate. Yang et al. (SUPERB) is the structural template; the shared-backbone, re-fit-head design of Evaluator follows it directly.

  • Reproducible scoring. Mattson et al. (MLPerf) is the auditable-run precedent; the RunManifest the harness writes on every run is the run-level provenance contract.

(Severity: low) No new literature entries are required for this report. The SUPERB “lightweight readout” contract is not yet numerically pinned in EMMA; that is an open recipe question, recorded in section 6, not a citation gap.

5. Experimental methodology and planned tests

Tests are written so an implementer can port them directly to tests/validation/harness/ once src/emma/ lands. Each entry records the falsifiable claim, the failure mode, the design, the sample size, the statistical test, the tolerance, and the gold-standard reference. Until src/emma/ exists, every test in this section is gated to skip-with-rationale rather than collect-as-failure; the active lens emits the plan plus the literature evidence, not a green check.

Transfer-overfit test - tests/validation/harness/test_transfer_overfit.py

  • Falsifiable claim. A backbone that overfit the training environment scores no better than the per-task chance baseline on the held-out environment.

  • Failure mode. An overfit model still scores above chance on the held-out environment, meaning the fold boundary leaks or the held-out environment is not a genuine transfer test.

  • Design. Train a backbone to convergence on the training environment only; run Evaluator under LEAVE_ONE_ENVIRONMENT_OUT; compare the held-out score to the per-task chance baseline.

  • Sample size. The v0.1 fold sizes (pinned in the recipe); the overfit backbone is the treatment, the chance baseline is the reference.

  • Statistical test. One-sided test that the overfit model does not beat the chance baseline on the held-out environment (paired bootstrap 95% confidence interval on per-scene error).

  • Tolerance. The overfit model’s held-out score within the chance baseline’s CI; a real transfer model must beat it one-sided.

  • Gold-standard reference. The per-task chance baseline (mean-angle predictor for E-LOC-AOA, prior-class predictor for the classifiers).

Split-guard test - tests/validation/harness/test_split_guard.py

  • Falsifiable claim. The harness detects a swapped train and eval split rather than silently scoring a memorization run.

  • Failure mode. A run that trains on the held-out environment and evaluates on the training environment posts a high score with no error.

  • Design. Construct a ProtocolRunner with the train and eval folds deliberately swapped; assert it raises SplitNotAvailable or a guard assertion, rather than producing a score.

  • Sample size. One fold pair per environment.

  • Statistical test. Assertion-based; the guard must fire, not score.

  • Tolerance. Zero tolerance; a swapped split must never produce a posted score.

  • Gold-standard reference. The Split assignments in the DataReleaseManifest.

Seed-determinism test - tests/validation/harness/test_seed_determinism.py

  • Falsifiable claim. The same backbone, recipe, and seed reproduce the same OOD average across runs.

  • Failure mode. Non-determinism in readout-head fitting or fold construction makes scores unreproducible.

  • Design. Run Evaluator twice with the same seed; assert the two RunManifest task_metrics match bit-for-bit, the RunManifest seed is recorded, and the per-task prediction_bundle_hashes match (so the scored outputs, not just the scores, are reproducible).

  • Sample size. Two runs per task.

  • Statistical test. Exact equality within floating-point tolerance.

  • Tolerance. Relative tolerance of 1e-6 on each metric; exact equality on prediction_bundle_hashes.

  • Gold-standard reference. The seed pinned in the manifest.

Run-manifest provenance test - tests/validation/harness/test_run_manifest_provenance.py

  • Falsifiable claim. Every run emits a RunManifest whose fields pin the auditable result triple: model provenance, data release, and per-task scored outputs.

  • Failure mode. A run posts a score with a manifest missing prediction_bundle_hashes, data_release_hash, or seed, so a reviewer cannot re-load exactly what was scored.

  • Design. Run Evaluator on a stub task; assert the returned EvalRun carries a RunManifest with model_provenance, data_release_hash, prediction_bundle_hashes: dict[str, str] keyed by TaskID value, task_metrics, seed, and harness_version all populated and non-empty.

  • Sample size. One run per task.

  • Statistical test. Assertion-based; every required field populated.

  • Tolerance. Zero tolerance; a missing field must fail the run, not default silently.

  • Gold-standard reference. The RunManifest field contract in Schemas.

Aggregation-sensitivity probe - tests/validation/harness/test_aggregation_sensitivity.py

  • Falsifiable claim. The OOD average that OODAvg (the sole Aggregator subclass) produces is not dominated by a single outlier task.

  • Failure mode. Removing one task and re-normalizing swings the OOD average by a margin larger than the ranking gap between adjacent models.

  • Design. A leave-one-task-out jackknife over the four scored tasks, re-applying OODAvg each time with the same frozen anchors; report the min and max OOD average across the four leave-one-out aggregates.

  • Sample size. The four v0.1 scored tasks.

  • Statistical test. The range (max minus min) of the jackknifed aggregates, compared to the inter-model ranking gap.

  • Tolerance. The jackknife range below the typical inter-model gap; a range above it flags a fragile aggregate.

  • Gold-standard reference. The frozen anchors recorded in the release manifest.

Open methodology question

(Severity: high, architectural, deferred to open questions) Leave-one-environment-out at v0.1 has \(N = 2\) environments (E_URBAN, E_RURAL). The environment-level statistic is the mean of two values, so its standard error is \(s / \sqrt{2}\) with one degree of freedom: essentially no power to distinguish models at the environment stratum, and a single outlier environment dominates the aggregate. The v0.1 leaderboard must therefore be read as a comparative ranking under a fixed protocol, not a population estimate of transfer, and every aggregate must ship with its per-environment components. The planned fix is more environments at v0.2+, raising \(N\) toward a statistically powered environment-level mean. This is the same finding recorded in Tasks validation section 5 and OOD protocol, restated here because the harness is where the underpowered mean is computed.

6. Robustness boundaries

Each boundary is a claim about where the validity breaks, paired with a proposed probe.

  • Fold leakage. (Severity: high) A single scene that crosses the train or eval boundary silently inflates the score. Probe: the split-guard test in section 5; additionally, a scene-id-level audit that the eval fold’s scene ids are disjoint from the training fold’s.

  • Seed non-determinism. (Severity: medium) If readout-head fitting is not fully seeded, scores drift across runs and the RunManifest seed field is meaningless and the per-task prediction_bundle_hashes no longer pin a reproducible output. Probe: the seed-determinism test in section 5.

  • Anchor leakage. (Severity: high) If the per-task frozen anchors were ever derived from holdout scores, or recomputed per submission, the OODAvg normalization stops being anti-gaming: a submitter could move the anchors to flatter its own ranks, or the anchors would encode information leaked from the eval fold. Probe: a contract test that anchors are read from the frozen DataReleaseManifest only, never from a submission or a holdout metric computation, matching the Aggregation anti-gaming property.

  • Protocol mis-configuration. (Severity: high) A task wired to the wrong OOD axis (for example E-ID-FP scored under OODAxis.ENVIRONMENT instead of OODAxis.DEVICE) measures the wrong transfer and is invisible without a guard. Probe: a contract test that each TaskSpec ood_axis matches the OODProtocol axis the runner applies, cross-checked against Task reference.

  • Small-N environment statistics. (Severity: high) The two-environment aggregate is fragile; covered in section 5. Probe: a leave-one-environment-out jackknife (drop each environment in turn) to bound the aggregate’s sensitivity to any single environment.

  • Aggregation fragility. (Severity: medium) The OOD average over four tasks can be dominated by one outlier task if its frozen anchor pair is poorly chosen. Probe: the aggregation-sensitivity jackknife in section 5; the min/max anchor sensitivity bounds the swing.

  • Readout-head capacity drift. (Severity: medium, deferred to open questions) The “lightweight readout” contract is not yet numerically pinned; a submitter could in principle use a heavy head that memorizes the training fold. Probe: pin the head architecture and capacity per task in the spec; reject a run whose head exceeds the pinned capacity.

7. Synthesis

Load-bearing claims supported. The shared-backbone, re-fit-head, leave-one-X-out design is construct-valid and follows the SUPERB and WILDS precedents. The fold-reduction mean is the right unit for a transfer score, and OODAvg is the sole Aggregator subclass that produces it; SimToRealGap correctly stays off that code path as a standalone reporter. The RunManifest provenance, now carrying per-task prediction_bundle_hashes, makes every run reproducible from its recorded triple: model provenance, data release hash, and the exact scored outputs.

Gaps surfaced. (1) Leave-one-environment-out at \(N = 2\) is underpowered for an environment-level statistic; the v0.1 aggregate is a comparative ranking, not a population estimate. (2) The fold boundary is a silent-failure surface; it needs the split-guard test rather than code inspection. (3) The readout-head capacity is not pinned, leaving the “lightweight readout” contract un-enforced. (4) The OOD average over four tasks is sensitive to a single outlier task until the jackknife bounds are measured. (5) The frozen-anchor anti-gaming property is asserted by contract; the anchor-leakage probe (anchors read from the release manifest, never the submission or the holdout metric) is the proposed enforcement.

Recommended actions. (1) Land the split-guard test as the first tests/validation/harness/ test; it is the cheapest insurance against the silent-failure surface. (2) Ship every v0.1 aggregate with its per-environment components, never the aggregate alone. (3) Add a third environment to the v0.2 recipe to raise \(N\) above two. (4) Pin the readout-head architecture and capacity per task in the TaskSpec before the leaderboard goes live. (5) Land the run-manifest provenance test alongside the seed-determinism test so the prediction_bundle_hashes pin is enforced from the first runnable release.

Top three findings.

  1. (high) Two-environment leave-one-environment-out is underpowered; the v0.1 aggregate is a comparative ranking under a fixed protocol, not a population estimate of transfer, and must ship with per-environment scores.

  2. (high) The fold boundary is a silent-failure surface; the split-guard test (swapped train and eval split must raise, not score) is the proposed first verification.

  3. (medium, architectural) The readout-head capacity is not pinned, so the “lightweight readout” contract that keeps the score a reflection of representation quality is not yet enforced.

References

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. The leave-one-X-out framing the protocol runner implements.

  • Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. One frozen backbone, lightweight readouts, one aggregate; the structural template for Evaluator. (verify)

  • Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Reproducible scoring with recorded provenance; the run-manifest contract.

  • Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” NeurIPS 2019, arXiv:1906.02530. Predictive performance under controlled shift. (verify)

See Also