Datasets and rfgen

Pre-launch / design spec

The recipe schema and rfgen config group references below are proposed contracts. They finalize when the dataset loader and the pinned rfgen release land.

Overview

EMMA (Electromagnetic Multi-task Model Assessment) consumes data; it does not generate it. The primary source is rfgen, Superpose’s synthetic radio-frequency (RF) scene generator, which produces raw, phase-coherent, multi-antenna I/Q (in-phase and quadrature) scenes from a frozen commit, configuration, and seed. Real-capture OOD (out-of-distribution) subsets come from public RF testbeds (Colosseum, POWDER, COSMOS). EMMA’s job is to pin exactly which rfgen output a task runs on, turn rfgen’s annotations into task labels, and make the result re-derivable. That pinning is a dataset recipe.

Place In The System

A dataset recipe is the contract between rfgen (which owns signal generation) and the EMMA pipeline (which owns evaluation). It is the artifact a dataset page documents, the input to EMMADataset, and the unit of content-hashed reproducibility. See Architecture for where the recipe sits in the pipeline and Reproducibility for how it is hashed.

Boundaries

  • EMMA owns: the recipe (a pinned rfgen configuration), the loader (EMMADataset), label extraction (LabelExtractor), and the real-capture adapters (RealCaptureAdapter and its implementations).

  • rfgen owns: emitters, channels, propagation, antenna and array geometry, scene composition, and on-disk storage. EMMA references rfgen’s config schema by name; it does not redefine emitter or channel types.

Data Flow

rfgen GenerationConfig            LabelExtractor
(commit + config + seed)  ──►  EMMADataset  ──►  (raw I/Q, task labels)
        │                                              │
        ▼                                              ▼
  content hash                                  per-task splits
  (auditable)                                   (train / dev / holdout)

What a Dataset Recipe Is

A dataset recipe is a frozen rfgen GenerationConfig plus a label-extraction map. Concretely, it pins:

  1. The rfgen commit. A single frozen git SHA. The generator, not a static file, is the test set.

  2. The resolved config. A fully composed rfgen configuration, naming the config groups below (emitter zoo, channel, scene, placement, label, storage, executor, run).

  3. Seed ranges. The integer seeds that parameterize scene generation, partitioned into splits.

  4. Split assignment. Which seeds are train, which are dev, and which are holdout. The holdout seeds are never published.

  5. A label-extraction map. A specification of how LabelExtractor turns rfgen metadata into the per-task labels a task trains on.

The recipe is recorded as a DatasetRecipe schema and summarized in a SceneManifest per scene. The full schema is documented at Dataset recipe.

rfgen Config Groups

EMMA references rfgen’s configuration groups by name; it does not redefine the RF physics inside them. The groups a recipe pins are:

  • emitter_zoo: the emitters in the scene (waveforms, drone-RF signatures, radar signals).

  • channel: the propagation environment (urban multipath, rural near-line-of-sight).

  • scene: scene composition, emitter counts, geometry.

  • placement: emitter and receiver placement.

  • label: the on-disk annotations rfgen writes.

  • storage: the on-disk format (SigMF, the Signal Metadata Format, and Zarr).

  • executor: how scenes are generated in parallel.

  • run: run-level parameters (seed ranges, counts).

The receiver array is specified through rfgen’s ArrayGeometry enum, whose members include ULA (uniform linear array, for azimuth-only tasks) and URA (uniform rectangular array, when elevation and azimuth both matter). EMMA reads this geometry into ArraySpec at load time.

Real-Capture OOD Subsets

Because synthetic-only benchmarks inherit their generator’s blind spots, every synthetic scene set is paired with a real-capture OOD subset (EMMA-REAL-OOD). Real captures are ingested through the RealCaptureAdapter ABC, with concrete implementations for the three public testbeds: ColosseumAdapter, PowderAdapter, and CosmosAdapter. These subsets feed the sim-to-real gap column, not the aggregate score. See Sim-to-real gap.

Minimal Example

A recipe is a pinned configuration, not a script you run to generate physics:

# dataset recipe (proposal) - pins rfgen output, does not specify RF physics
dataset_id: EMMA-LOC-v0.1
rfgen:
  commit: a91f3c2
  config: scenes/v0.1/urban-mimo.yaml
  array_geometry: URA
  num_rx: 8
splits:
  train:   {start: 1, stop: 160000, count: 160000}
  dev:     {start: 160001, stop: 200000, count: 40000}
  holdout: {start: 200001, stop: 240000, count: 40000}   # secret; never published
label_map:
  E_LOC_AOA: {field: emitters[].aoa_deg, kind: regression}
  E_LOC_LOS: {field: los_flag, kind: binary}

Design Notes

  • The generator, not a static file, is the test set. A bug is fixed by shipping a new content-hashed release without invalidating provenance. This fixes the failure modes that bit static-file benchmarks: frozen label errors, dying eval servers, and leaking test sets. See Reproducibility.

  • EMMA never specifies RF physics. If a recipe appears to define an emitter, a channel, or a propagation model, it has crossed the boundary into rfgen. The recipe only pins and labels rfgen output.

  • One recipe, many labels. A recipe targets one scene set but can carry a label map for several tasks, because one scene yields many labels via LabelExtractor. See Data model.

References

  • NVIDIA Sionna: Hoydis et al., “Sionna: An Open-Source Library for Next-Generation Physical Layer Research,” 2023, arXiv:2203.11854. The ray-tracing and link-level simulator rfgen builds on.

  • Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. The shared-input, many-readouts contract a recipe serves.

See Also