Datasets & scenes

Pre-launch / design spec

The registry, environment labels, and split sizes below are proposals. EMMA is not yet released; nothing here is downloadable yet.

EMMA’s scenes are generated, not captured - produced by rfgen, Superpose’s synthetic RF (radio-frequency) scene generator. The canonical input is raw complex multi-antenna I/Q (in-phase and quadrature, SigMF), which preserves the inter-antenna phase coherence that localization and beam tasks depend on. Derived views (spectrograms, CSI: channel state information) are offered as secondary, never primary.

Because synthetic-only benchmarks inherit their generator’s blind spots, every synthetic scene set is paired with a real-capture OOD (out-of-distribution) validation subset, and the leaderboard reports a sim-to-real gap alongside every synthetic score.

Reproducible by construction

A scene is fully determined by a frozen rfgen commit, a config, and a seed. EMMA content-hashes that triple, so anyone can re-derive the benchmark bit-for-bit - the generator, not a static file, is the test set. This fixes the failure modes that bit every static-file benchmark: label errors frozen into a release (RadioML’s admitted errata), eval servers that die, and test sets that leak. A bug is fixed by a new hashed release without invalidating provenance.

# example scene manifest (proposal)
scene_id: emma-scene-00017a
generator: rfgen
commit: a91f3c2
config: scenes/v0.1/urban-mimo.yaml
seed: 17
content_hash: sha256:b7e3...f019
arrays: [rx-8elem-ura]
emitters:
  - {type: drone-rf, waveform: ofdm, aoa_deg: 42}
  - {type: comms,    waveform: qpsk,  aoa_deg: 118}
channel: urban-multipath
snr_db: 12.0

Each scene’s full provenance record is a SceneManifest, and the frozen configuration behind a whole dataset is a DatasetRecipe. See the Dataset recipe schema for the exact contract.

Environments & the OOD axis

Generalization is the score, so the environment structure is the benchmark. v0.1 ships with at least two synthetic channel environments so leave-one-environment-out is live from day one. Environment identifiers are Environment members; the full catalog and the per-task OODAxis members live in Environment reference.

Env ID

Description

Used for

E_URBAN

Dense urban multipath, sub-6 GHz (Sionna UMa)

train + one OOD target

E_RURAL

Sparse rural / near-LoS, sub-6 GHz (Sionna RMa or TDL near-LoS)

train + one OOD target

E_REAL_CAP

Real captures (Colosseum / POWDER / COSMOS)

sim-to-real gap column

Additional axes (unseen device, unseen frequency band, unseen SNR regime) are layered per task - see Task reference.

Splits

Split

Purpose

Released

train

Pretraining / fine-tuning

public (regenerable)

dev

Local evaluation, prototyping, public leaderboard proxies

public (regenerable)

holdout

Official scoring

private - never released

The holdout is a frozen secret seed. Scores on it are produced by prediction submission at v0.1 (you submit predictions; EMMA scores them) and by sandboxed code submission at v1. Superpose’s own model is re-scored on the holdout by a third party on every release.

Registry

The authoritative catalog of EMMA datasets. One row per dataset id. The Task column links the TaskID each recipe feeds; Version is the release badge; Maturity is the contract state (every recipe is proposed-contract until src/emma/ lands). Recipes that depend on an rfgen feature not yet shipped carry a flag in their per-dataset page.

Dataset ID

Task

Antennas

Carrier / BW

Source

Split

Version

Maturity

EMMA-LOC-v0.1

E-LOC-AOA, E-LOC-LOS

8-elem URA

sub-6 / 20 MHz

rfgen

train/dev/holdout

v0.1

proposed-contract

EMMA-DRONE-v0.1

E-ID-DRONE

8-elem ULA

2.4 / 5.8 GHz

rfgen

train/dev/holdout

v0.1

proposed-contract

EMMA-FP-v0.1

E-ID-FP

8-elem ULA

sub-6 / 20 MHz

rfgen

leave-one-unit-out

v0.1

proposed-contract

EMMA-AMC-v0.1

E-ID-AMC

8-elem (SISO-compatible)

sub-6

rfgen

train/dev/holdout

v0.1

proposed-contract

EMMA-REAL-OOD-v0.1

fidelity (sim-to-real gap)

4 to 8-elem

various

Colosseum / POWDER / COSMOS

dev only

v0.1

proposed-contract

EMMA-RWAVE-v0.2

E-ID-RWAVE

8-elem ULA

radar bands

rfgen

train/dev/holdout

v0.2

proposed-contract

EMMA-UAVDOP-v0.2

E-ID-UAVDOP

8-elem ULA

radar bands

rfgen

train/dev/holdout

v0.2

proposed-contract

EMMA-BEAM-v0.2

E-CH-BEAM

32 / 64-elem massive MIMO

5G NR

rfgen

train/dev/holdout

v0.2

proposed-contract

EMMA-POS-v1

E-LOC-POS

8-elem URA

sub-6 / 20 MHz

rfgen

train/dev/holdout

v1

proposed-contract

EMMA-CSI-v1

E-CH-CSI

32 / 64-elem massive MIMO

5G NR

rfgen

train/dev/holdout

v1

proposed-contract

EMMA-SC-SEP-v1

E-SC-SEP

8-elem ULA

wideband

rfgen

train/dev/holdout

v1

proposed-contract

EMMA-SC-ANOM-v1

E-SC-ANOM

8-elem ULA

wideband

rfgen

train/dev/holdout

v1

proposed-contract

EMMA-SC-SENSE-v1

E-SC-SENSE

8-elem ULA

wideband

rfgen

train/dev/holdout

v1

proposed-contract

EMMA-S2T-CAP-v2

E-S2T-CAP

8-elem ULA

wideband

rfgen + annotator

train/dev/holdout

v2

proposed-contract

EMMA-S2T-QA-v2

E-S2T-QA

8-elem ULA

wideband

rfgen + annotator

train/dev/holdout

v2

proposed-contract

Note

E-ID-AMC (EMMA-AMC-v0.1) is a continuity column: reported but excluded from the aggregated OOD-avg score. EMMA-REAL-OOD-v0.1 is fidelity only and never enters the aggregate.

Why multi-antenna I/Q is non-negotiable

AoA/DoA (angle/direction-of-arrival) and beam tasks read the phase difference between antennas - that coherence is destroyed the moment you take a magnitude spectrogram or reduce to CSI. Every existing RF dataset is either single-antenna (RadioML, TorchSig) or a derived representation (DeepMIMO at CSI level; every radar dataset’s CFAR point cloud). EMMA’s input is the layer the field is converging toward but no incumbent dataset natively provides.

Regenerating a dataset

Each dataset is generated by a pinned rfgen recipe (commit, emitters, array, channel, label extraction, seeds). The per-dataset recipe pages and the regeneration procedure are in the developer documentation.

Datasets