Datasets & scenes¶
Pre-launch / design spec
The registry, environment labels, and split sizes below are proposals. EMMA is not yet released; nothing here is downloadable yet.
EMMA’s scenes are generated, not captured - produced by
rfgen, Superpose’s
synthetic RF (radio-frequency) scene generator. The canonical input is raw
complex multi-antenna I/Q (in-phase and quadrature, SigMF), which preserves the
inter-antenna phase coherence that localization and beam tasks depend on.
Derived views (spectrograms, CSI: channel state information) are offered as
secondary, never primary.
Because synthetic-only benchmarks inherit their generator’s blind spots, every synthetic scene set is paired with a real-capture OOD (out-of-distribution) validation subset, and the leaderboard reports a sim-to-real gap alongside every synthetic score.
Reproducible by construction¶
A scene is fully determined by a frozen rfgen commit, a config, and a seed.
EMMA content-hashes that triple, so anyone can re-derive the benchmark
bit-for-bit - the generator, not a static file, is the test set. This fixes
the failure modes that bit every static-file benchmark: label errors frozen into
a release (RadioML’s admitted errata), eval servers that die, and test sets that
leak. A bug is fixed by a new hashed release without invalidating provenance.
# example scene manifest (proposal)
scene_id: emma-scene-00017a
generator: rfgen
commit: a91f3c2
config: scenes/v0.1/urban-mimo.yaml
seed: 17
content_hash: sha256:b7e3...f019
arrays: [rx-8elem-ura]
emitters:
- {type: drone-rf, waveform: ofdm, aoa_deg: 42}
- {type: comms, waveform: qpsk, aoa_deg: 118}
channel: urban-multipath
snr_db: 12.0
Each scene’s full provenance record is a SceneManifest, and the frozen configuration behind a whole dataset is a DatasetRecipe. See the Dataset recipe schema for the exact contract.
Environments & the OOD axis¶
Generalization is the score, so the environment structure is the benchmark. v0.1 ships with at least two synthetic channel environments so leave-one-environment-out is live from day one. Environment identifiers are Environment members; the full catalog and the per-task OODAxis members live in Environment reference.
Env ID |
Description |
Used for |
|---|---|---|
|
Dense urban multipath, sub-6 GHz (Sionna UMa) |
train + one OOD target |
|
Sparse rural / near-LoS, sub-6 GHz (Sionna RMa or TDL near-LoS) |
train + one OOD target |
|
Real captures (Colosseum / POWDER / COSMOS) |
sim-to-real gap column |
Additional axes (unseen device, unseen frequency band, unseen SNR regime) are layered per task - see Task reference.
Splits¶
Split |
Purpose |
Released |
|---|---|---|
|
Pretraining / fine-tuning |
public (regenerable) |
|
Local evaluation, prototyping, public leaderboard proxies |
public (regenerable) |
|
Official scoring |
private - never released |
The holdout is a frozen secret seed. Scores on it are produced by prediction submission at v0.1 (you submit predictions; EMMA scores them) and by sandboxed code submission at v1. Superpose’s own model is re-scored on the holdout by a third party on every release.
Registry¶
The authoritative catalog of EMMA datasets. One row per dataset id. The Task column links the TaskID each recipe feeds; Version is the release badge; Maturity is the contract state (every recipe is proposed-contract until src/emma/ lands). Recipes that depend on an rfgen feature not yet shipped carry a flag in their per-dataset page.
Dataset ID |
Task |
Antennas |
Carrier / BW |
Source |
Split |
Version |
Maturity |
|---|---|---|---|---|---|---|---|
|
|
8-elem URA |
sub-6 / 20 MHz |
rfgen |
train/dev/holdout |
v0.1 |
|
|
|
8-elem ULA |
2.4 / 5.8 GHz |
rfgen |
train/dev/holdout |
v0.1 |
|
|
|
8-elem ULA |
sub-6 / 20 MHz |
rfgen |
leave-one-unit-out |
v0.1 |
|
|
|
8-elem (SISO-compatible) |
sub-6 |
rfgen |
train/dev/holdout |
v0.1 |
|
|
fidelity (sim-to-real gap) |
4 to 8-elem |
various |
Colosseum / POWDER / COSMOS |
dev only |
v0.1 |
|
|
|
8-elem ULA |
radar bands |
rfgen |
train/dev/holdout |
v0.2 |
|
|
|
8-elem ULA |
radar bands |
rfgen |
train/dev/holdout |
v0.2 |
|
|
|
32 / 64-elem massive MIMO |
5G NR |
rfgen |
train/dev/holdout |
v0.2 |
|
|
|
8-elem URA |
sub-6 / 20 MHz |
rfgen |
train/dev/holdout |
v1 |
|
|
|
32 / 64-elem massive MIMO |
5G NR |
rfgen |
train/dev/holdout |
v1 |
|
|
|
8-elem ULA |
wideband |
rfgen |
train/dev/holdout |
v1 |
|
|
|
8-elem ULA |
wideband |
rfgen |
train/dev/holdout |
v1 |
|
|
|
8-elem ULA |
wideband |
rfgen |
train/dev/holdout |
v1 |
|
|
|
8-elem ULA |
wideband |
rfgen + annotator |
train/dev/holdout |
v2 |
|
|
|
8-elem ULA |
wideband |
rfgen + annotator |
train/dev/holdout |
v2 |
|
Note
E-ID-AMC (EMMA-AMC-v0.1) is a continuity column: reported but excluded from the aggregated OOD-avg score. EMMA-REAL-OOD-v0.1 is fidelity only and never enters the aggregate.
Why multi-antenna I/Q is non-negotiable
AoA/DoA (angle/direction-of-arrival) and beam tasks read the phase difference between antennas - that coherence is destroyed the moment you take a magnitude spectrogram or reduce to CSI. Every existing RF dataset is either single-antenna (RadioML, TorchSig) or a derived representation (DeepMIMO at CSI level; every radar dataset’s CFAR point cloud). EMMA’s input is the layer the field is converging toward but no incumbent dataset natively provides.
Regenerating a dataset¶
Each dataset is generated by a pinned rfgen recipe (commit, emitters, array, channel, label extraction, seeds). The per-dataset recipe pages and the regeneration procedure are in the developer documentation.