Open questions

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

Unresolved decisions that gate the credibility or coverage of EMMA (Electromagnetic Multi-task Model Assessment). Each item states the context, what is at stake, and what would resolve it. A question is listed here rather than asserted as fact because no primary source or implemented test yet settles it. Items that turn into architectural decisions are deferred to the maturity matrix once resolved.

  1. Is the Northeastern co-steward secured? Northeastern University is the strongest candidate to physically hold the private test-set seed and provide the real-capture OOD (out-of-distribution) subset (Colosseum, POWDER, COSMOS). The relationship is outside Superpose’s control and is the schedule-critical path for v0.1, because it solves governance and the real-capture credibility problem in a single step. Resolved by a signed co-steward agreement with a named external party and a published re-scoring procedure.

  2. Does IQFM submit early? IQFM is the one existing raw I/Q (in-phase and quadrature) foundation model, so its submission is the cleanest external test of whether EMMA’s surface is legible to the camp it most resembles. Resolved by an external IQFM (or comparable raw-I/Q model) submission within the first submission windows.

  3. Is the seed backbone honestly beatable? Visible vulnerability is the credibility signal. The seeded baseline is designed to be mid-tier and to lose publicly on at least one real-data column early. If an operator seed sweeps every column, the board reads as marketing and the ring-fence never earns trust. Resolved by an external model beating the seed on a real-data column within roughly the first six months of leaderboard operation.

  4. Does rfgen preserve multi-antenna phase coherence at the fidelity localization and beam require? Inter-antenna phase coherence is the load-bearing property for E-LOC-AOA and E-CH-BEAM. If rfgen’s channel chain introduces phase errors large enough to dominate the angular-error metric, the localization construct is invalid regardless of the protocol. Resolved by a side-by-side comparison of rfgen-generated array phase against a real captured array, with a stated tolerance, once the recipe and a real capture exist. See Datasets validation.

  5. Does the CSI (channel state information) camp adopt the bridge? E-CH-CSI is offered as a bridge task so CSI-native models (LWM, DeepMIMO) have an on-ramp. Adoption is uncertain because that camp chose a different input modality and may not submit to an I/Q bench. Resolved by at least one CSI-camp model entering the v1 track.

  6. How is the signal-to-text task defined from scratch? No community does RF (radio-frequency) captioning or question-answering at scale, so there is no established label schema, metric anchor, or task granularity to inherit. E-S2T-CAP and E-S2T-QA therefore ship at v2, after the inner ring is proven. Resolved by a concrete label-extraction map, a caption or QA gold standard, and a construct-validity pass in Tasks validation.

  7. What is the go or no-go signal? The venture-deciding test: within the first two submission windows, at least one external model must beat the seed on at least one real-data column. That single event simultaneously falsifies the sim-to-real critique (real scores are reported and someone generalized) and the neutrality critique (the operator lost publicly). If it does not fire within roughly six months of launch, the benchmark is indistinguishable from a self-reported board and must be re-scoped. Resolved by the leaderboard record itself.

  8. Does the OOD protocol have statistical power at only two environments? v0.1 ships with at least two synthetic channel environments (E_URBAN, E_RURAL). A leave-one-environment-out mean over two folds has wide variance, so a small per-environment gap may be noise rather than signal. Resolved by a power analysis on the per-fold variance, and by adding environments in later releases. See OOD protocol.

  9. Which rfgen emitter and channel coverage do the radar and drone recipes require? E-ID-RWAVE, E-ID-UAVDOP, and E-ID-DRONE depend on rfgen emitters (radar waveforms, drone-RF signatures) and channel environments that rfgen v0 may not yet ship. A recipe that references an unshipped emitter is incomplete, not merely future. Resolved by a documented rfgen coverage map per recipe, with unshipped dependencies flagged until they land. See Datasets and rfgen.

Resolved

  • Canonical direction-of-arrival metadata path (was #10). Resolved 2026-07-18. Verified against rfgen (src/rfgen/propagation.py, the label schema, the annotators, and the audit all use aoa_deg): the canonical per-emitter arrival-angle field is aoa_deg. Every doa_deg, doa_azimuth_deg, doa_elevation_deg, and emitter_doa_deg in the docs is swept to emitters[].aoa_deg, collapsing the two-field azimuth/elevation map to a single aoa_deg entry. v0.1 E-LOC-AOA scores the scalar arrival azimuth; elevation extends the target at v1.

  • SimToRealGap and the Aggregator ABC (was #11). Resolved 2026-07-18. SimToRealGap takes a pairwise (synth_score, real_score) signature that is not substitutable for the Aggregator __call__(fold_scores) contract, so inheriting it is a Liskov violation. SimToRealGap is now a standalone fidelity-gap reporter; OODAvg remains the sole Aggregator.

  • Captioning-metric acronym convention (was #12). Resolved 2026-07-18. BLEU (Bilingual Evaluation Understudy), METEOR (Metric for Evaluation of Translation with Explicit ORdering), and CIDEr (Consensus-based Image Description Evaluation) are expanded on first use on every page, treated as acronyms rather than proper nouns, consistent with STYLE.md.

  • neutrality.md title versus link text (was #13). Resolved 2026-07-18. The page H1 is renamed to “Neutrality” and every inbound link’s text is swept to “Neutrality”, matching the filename.

  • Helper types in the schemas reference (was #14). Resolved 2026-07-18 by adding a “Helper types” subsection to Schemas defining SeedRange, MetadataPath, and EmitterRecord so an implementer can code against them.

  • LeaderboardRow audit-triple wording (was #15). Resolved 2026-07-18 by softening the LeaderboardRow prose in Schemas: the row carries the model display name, the data release hash, and the per-task scores, and points to the RunManifest for the prediction-bundle hash that pins exactly which outputs were scored.

  • F1 in the v0.1 metrics scope (was #16). Resolved 2026-07-18 by dropping F1 from the E-ID-DRONE task contract. AUC is the single v0.1 metric for drone detection, matching the five-metric v0.1 scope in Metrics validation; the F1 class stays scoped to E-S2T-QA.

  • RadioML publisher attribution (was #17). Resolved 2026-07-18 by re-attributing the claim to DeepSig’s datasets page (deepsig.ai/datasets) and standardizing the wording to “its publisher states the data has known errata and is not used in its products” wherever the AMC continuity column is discussed.

See Also

  • Governance: the co-steward and ring-fence questions in policy form.

  • Validation methodology: how a documented inability to test becomes a tracked gap rather than a silent risk.

  • Maturity matrix: where resolved items land.

  • Roadmap: the release sequence these questions gate.

  • Landscape: the incumbents these questions are measured against.