Validation background

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

EMMA (Electromagnetic Multi-task Model Assessment) is scientific work, so each task, metric, protocol, and dataset recipe is audited before it is asserted. This page is the short pointer from the Background section to the Validation section where the per-layer reports live. The framework itself is documented in Validation methodology.

The six-lens framework

Each lens asks one question no other lens asks. The construct-validity lens carries the extra responsibility of checking domain utility, not only internal validity.

  • Construct validity. Does each task and OOD (out-of-distribution) axis measure what it claims, at the level of abstraction its target domain needs, and is the capability one that domain actually cares about?

  • Mathematical fidelity. Is every metric formula correct in sign, units, and normalization against its cited definition?

  • Empirical realism. Is each pinned rfgen recipe physically credible for its domain, and does it match what real captures show?

  • Literature grounding. Is every load-bearing choice cited to a canonical, verifiable source, and does each metric reuse a standard library rather than a hand-rolled definition?

  • Experimental methodology. Is the benchmark’s own method sound: OOD protocol power, aggregation robustness, held-out overfitting resistance, split non-leakage?

  • Robustness boundaries. Where does the validity break: rfgen phase fidelity, metric gaming, small-N environment statistics, sim-to-real confounds?

Pre-implementation posture

EMMA has no src/emma/ yet, so three lenses (empirical realism, experimental methodology, robustness boundaries) produce proposed evidence rather than test results: the test design, the sample size, the statistical test, the tolerance, and the gold-standard reference that will run when the harness exists. A documented inability to test a claim today is tracked as a gap, not hidden; unresolved items surface in Open questions.

Per-layer report catalog

The five load-bearing layers (datasets, metrics, tasks, harness, leaderboard) carry full six-lens reports; the four engineering layers (primitives, schemas, CLI, governance) carry construct, literature, and methodology sections. The catalog and the report shape live in the Validation index.

See Also