Architecture

Pre-launch / design spec

The pipeline below is the design target for v0.1. Class names are proposed contracts; signatures land with the harness implementation.

Overview

EMMA (Electromagnetic Multi-task Model Assessment) scores one frozen electromagnetic foundation model across many tasks, under controlled out-of-distribution (OOD) axes, and aggregates the result into a single transfer score. The pipeline is linear: a frozen backbone produces embeddings, lightweight task-specific readout heads map those embeddings to outputs, an OOD protocol slices the evaluation into held-out folds, per-task metrics score each fold, and an aggregator rolls the folds into the headline number. The same backbone is then re-scored on a real-capture OOD subset so the leaderboard can report a sim-to-real gap alongside the synthetic score.

Place In The System

The pipeline sits between data (owned by rfgen and public radio-frequency (RF) testbeds) and reporting (the leaderboard). EMMA owns the middle: the task registry, the readout-head contracts, the OOD protocols, the metrics, the evaluator that drives them, and the leaderboard store that records results. It does not own signal generation. See Datasets and rfgen for the boundary.

Boundaries

Data Flow

The pipeline runs in seven stages. Each stage corresponds to a concrete EMMA abstraction.

[1] rfgen scenes ───────► [2] EMMADataset ──────► [3] FrozenBackbone
    commit + config + seed    + LabelExtractor        (weights frozen)
    + real-capture adapters     raw multi-antenna        │
                               I/Q + per-task labels     │
                                                         ▼
                                               [4] ReadoutHead(s)
                                                 one per task pillar
                                                         │
                 per-task OOD axis ◄──────────────────────┤
                 (leave-one-environment-out,              │
                  leave-one-unit-out, ...)                │
                                                         ▼
                                               [5] Metric(s)
                                                 per task, per fold
                                                         │
                                                         ▼
                                               [6] Aggregator
                                                 OODAvg + SimToRealGap
                                                         │
                                                         ▼
                                               [7] LeaderboardStore
                                                 private held-out scoring

  1. Scenes. rfgen produces synthetic scenes from a frozen (commit, config, seed) triple. Real-capture OOD scenes come from Colosseum, POWDER, and COSMOS via RealCaptureAdapter implementations. See Datasets and rfgen.

  2. Dataset. EMMADataset loads raw multi-antenna I/Q; LabelExtractor turns rfgen metadata into per-task labels. See Data model.

  3. Backbone. FrozenBackbone is the SUPERB (Speech processing Universal PERformance Benchmark) upstream contract: weights are frozen, only the readout trains. One backbone is scored across every task.

  4. Readout heads. A ReadoutHead per task (for example AngularRegressionHead for angle-of-arrival, ClassificationHead for line-of-sight versus non-line-of-sight). Heads are lightweight; the backbone carries the representational load.

  5. OOD protocol. OODProtocol slices each task into train and held-out folds along a controlled axis: LeaveOneEnvironmentOut for localization, LeaveOneUnitOut for device fingerprinting. The reported score is the mean over held-out folds. See Generalization.

  6. Metrics and aggregation. A Metric scores each fold (for example MeanAngularError). OODAvg normalizes per-task scores to a shared 0 to 1 range and averages them; SimToRealGap reports the difference between the synthetic and real-capture scores. See Sim-to-real gap.

  7. Leaderboard. LeaderboardStore records each row on the private held-out split, carrying the data-release hash and the per-task scores; the row links its RunManifest, which carries the per-task prediction-bundle hashes, so a result is an auditable triple. See Leaderboard.

Minimal Example

One backbone, scored across the four v0.1 aggregated tasks, each under its OOD axis:

from emma.enums import TaskID, ReleaseVersion
from emma.tasks import TaskRegistry, FrozenBackbone
from emma.harness import Evaluator

backbone = FrozenBackbone.from_pretrained("my-fm@v1")
evaluator = Evaluator(backbone=backbone, version=ReleaseVersion.V0_1)

for task_id in (TaskID.E_LOC_AOA, TaskID.E_LOC_LOS, TaskID.E_ID_DRONE, TaskID.E_ID_FP):
    task = TaskRegistry.get(task_id, version=ReleaseVersion.V0_1)
    run = evaluator.run(task)          # EvalRun: per-fold metrics + ood_avg
    print(task_id, run.ood_avg)

Design Notes

  • One backbone, many readouts. The Evaluator enforces that the same backbone instance feeds every task, so the score reflects representation quality, not head capacity. This is the SUPERB contract; see Signal as a modality.

  • The OOD axis is the contract. A task is not fully specified without its axis. LeaveOneEnvironmentOut and LeaveOneUnitOut answer different questions. See Generalization.

  • Aggregation before reporting. OODAvg and SimToRealGap are computed by their classes, never by hand. The leaderboard never shows a raw per-fold number as the headline.

  • Train and dev are re-derivable; the held-out is not. Only the held-out split is scored for the leaderboard, and only via prediction submission at v0.1. See Reproducibility.

References

  • Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. The one-frozen-backbone, many-lightweight-readouts, one-aggregated-score template this pipeline follows.

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. Leave-one-X-out as the evaluation contract rather than an appendix.

See Also