Validation methodology

Pre-launch / design spec

This page documents the validation framework. Per-layer reports under this section apply the framework to the proposed contracts and pinned rfgen recipes. The empirical tests they specify run once src/emma/ lands.

EMMA (Electromagnetic Multi-task Model Assessment) is scientific work. Its credibility rests on whether each task, metric, protocol, and dataset recipe actually measures what it claims, and whether the measurement is useful to the domain it targets. This section applies the six-lens academic-research-methods framework inherited from rfgen, adapted for a benchmark that is, by design, a documented specification rather than running code.

What is being validated

EMMA has no src/emma/ yet. The artifact under validation is the design: the proposed task, metric, and protocol contracts in the reference and API pages, and the pinned rfgen recipes in the datasets section. Validation asks whether these contracts are defensible before any code is written.

A benchmark’s two failure modes are specific. First, a task can be internally consistent but measure the wrong thing (construct drift). Second, a task can measure the right thing in a way no real community uses (domain irrelevance). The six lenses catch both.

The six lenses

Each lens asks one question no other lens asks. The construct-validity lens carries an extra responsibility: it checks domain utility, not only internal validity.

The six lenses and the question each asks of the EMMA design

Lens

Question

construct-validity

Does each task and OOD (out-of-distribution) axis measure what it claims, at the level of abstraction its target domain needs, and is the capability one that domain actually cares about?

mathematical-fidelity

Is every metric formula correct in sign, units, and normalization against its cited definition?

empirical-realism

Is each pinned rfgen recipe physically credible for its domain? Does it match what real captures show, and where does it not?

literature-grounding

Is every load-bearing choice cited to a canonical, verifiable source, and does each metric reuse a standard library rather than a hand-rolled definition?

experimental-methodology

Is the benchmark’s own method sound: OOD protocol power, aggregation robustness, held-out overfitting resistance, split non-leakage?

robustness-boundaries

Where does the validity break: rfgen phase fidelity, metric gaming, small-N environment statistics, sim-to-real confounds?

Construct validity and domain utility

Construct validity is the lens that audits the construct itself before any other check is meaningful. For EMMA it carries two questions.

Internal validity. Does the task measure the named property? E-LOC-AOA scored as continuous regression measures angular error; scoring it as binned classification would let a model hide systematic bias behind class accuracy, so the regression choice is part of the construct, not a presentation detail. E-ID-FP scored leave-one-unit-out measures whether a model recognizes an unseen device, not whether it memorized a capture session; the protocol is the construct.

Domain utility. A task can be internally valid yet useless to the community it claims to serve. Each task names a target domain and the capability that domain depends on. Radar waveform recognition serves defense emitter identification; multi-antenna drone detection serves defense and spectrum monitoring; localization serves communications positioning, defense, and RF (radio-frequency) drone handling; fingerprinting serves the DARPA RFMLS lineage; beam management serves 3GPP AI/ML. A task with no real buyer is a finding, not a feature.

A clean construct-validity pass records, per task: the claim, the implemented proxy (the label extraction and metric), known omissions, the downstream risk, and the domain that depends on the capability.

Pre-implementation posture

Three lenses (empirical-realism, experimental-methodology, robustness-boundaries) are “active” in rfgen, where they run pytest, generate figures, and compare to real captures. EMMA has no code, so these lenses produce proposed evidence: the test design, the sample size, the statistical test, the tolerance, and the gold-standard reference that will run when the harness exists. A documented inability to test a claim today is higher quality than a test that does not test what it claims.

The proposed tests are written so an implementer can turn them directly into tests/validation/<layer>/ once src/emma/ lands. At that point the ported scientific-validation-orchestrator skill runs the refine-until-clean loop against real code.

Report shape

Each report at validation/<layer>.md has seven sections, in order:

  1. Purpose and construct (construct-validity): the layer’s claim, its proxy, omissions, downstream risk, and domain utility.

  2. Mathematical fidelity (mathematical-fidelity): line-by-line audit of every formula, sign, unit, and constant against its citation.

  3. Empirical realism (empirical-realism): whether the data design matches real captures, with proposed side-by-side figures.

  4. Literature grounding (literature-grounding): the canonical source for every load-bearing element, and library reuse over hand-rolling.

  5. Experimental methodology and planned tests (experimental-methodology): the proposed experiments, sample sizes, statistical tests, tolerances, and gold-standard references.

  6. Robustness boundaries (robustness-boundaries): where the validity breaks, the operating envelope, and proposed sweep and probe tests.

  7. Synthesis: load-bearing claims supported, gaps surfaced, recommended actions, and the top three findings.

Depth scales with scientific load. The five load-bearing layers (datasets, metrics, tasks, harness, leaderboard) carry full six-lens reports. The four engineering layers (primitives, schemas, cli, governance) carry all seven sections but concentrate depth on the construct, literature, and methodology lenses; their mathematical-fidelity, empirical-realism, and robustness-boundaries sections are present but lighter because these layers carry no formulas and no data.

Severity and architectural deferrals

Findings carry a severity (high, medium, low) and an architectural flag. Architectural findings change public API, add a config surface, or require a scope decision; they are deferred to background/open-questions.md with a one-sentence rationale rather than silently resolved. A high-severity finding in a load-bearing layer is a gap the synthesis section names explicitly.

Library-first check

Before recommending any custom logic, a lens checks whether a standard library or an established benchmark already defines it. Metrics reuse torchmetrics, scipy, or numpy. Protocols reuse established benchmark precedents (SUPERB, WILDS). Dataset recipes reuse rfgen’s config schema rather than re-specifying emitter or channel physics. Hand-rolled logic that bypasses this check is a finding, even if it is functionally correct.

See Also