Principles¶

Standing instructions for any agent or contributor working in emma-benchmark. This file is the single source of truth for engineering and documentation principles. Writing-style rules (em-dashes, headings, anchors, badges, enum-vs-string examples) live in STYLE.md.

This file has two parts:

Part 1: First principles (stable). Timeless engineering and documentation posture, plus the four EMMA design pillars. These should not change as the design evolves.
Part 2: Current design contracts (mutable). The project’s concrete choices today: what EMMA owns versus what rfgen owns, the current boundary ABC list, smell-test items, and the done-checklist. These follow from Part 1 and will change as the design evolves.

When a documentation rule conflicts with a doc page, the rule wins until the doc is updated.

Part 1: First principles¶

These should not change as the design evolves.

Engineering principles¶

1. Calibrated humility¶

Reuse what is verified. Treat all sources (specs, docs, your own draft, cited papers) as fallible. Mark uncertainty explicitly. Pause when unsure. A merged change you cannot explain is technical debt at full interest.

Examples in this repo:

Default to rfgen for every dataset, and to standard libraries (PyTorch, torchmetrics, SciPy) for every metric. Do not reimplement a metric that torchmetrics already ships.
Cite every load-bearing claim to a primary source. Mark a claim proposed-contract or open question when no primary source is available, rather than presenting it as fact.
Treat the SABRE research digests and this design spec as fallible: verify a number before relying on it, and flag (verify) when you cannot.
Stop and ask before defining a new metric, a new OOD axis, or a new task from scratch. The right move is search-then-surface, not charge ahead.

2. Domain over generic¶

Adopt the domain’s established vocabulary (communications, radar, spectrum, RF fingerprinting) rather than reusing generic words that collide with RF or ML meanings. State responsibilities concretely; avoid vague architecture phrases that do not name the method, data, or behavior.

Examples in this repo:

Use leave-one-environment-out and leave-one-unit-out, not generic “cross-validation”, because the OOD axis is the contract.
Use angle-of-arrival and direction-of-arrival precisely; AoA is the receiver-side estimate of a signal’s direction, and binning it into classes hides systematic bias that regression exposes.
Before introducing a new term, check 3GPP, IEEE, ITU, and the major RF libraries (Sionna, TorchSig, rfgen).
Avoid “the benchmark owns / the harness exposes / the layer manages” without naming the concrete classes or data involved.

3. Encode rules; do not describe them¶

Code and machine-checkable schema are the contract; prose drifts. If a rule can be machine-checked, machine-check it.

Examples in this repo:

Closed sets of choices use StrEnum from emma.enums. Open plugin registries stay as str. YAML may use the enum string value because Pydantic deserializes it.
Task IDs, OOD axes, and metric directions are enums, not magic strings. A task that is not in TaskID cannot be scored.
Doc linting (scripts/docs_lint.py) fails the build on em-dashes, undefined acronyms, and unresolved {ref} anchors, so style violations cannot land silently.

4. Propagate, then close¶

A change is not done until its consequences are reflected everywhere. Half-applied edits leave the system inconsistent; the inconsistency is silent. Match scope to reviewability. Co-locate rationale with the decision. State silent gaps explicitly.

Examples in this repo:

After a task rename or a metric change, sweep all docs, schemas, recipes, anchors, and the link map in the same change.
Surface rationale on user-facing pages, not buried in background/. The reason modulation classification is a continuity column lives on the Tasks page where the decision is read.
“Looks done” is not a signal. When a change defers a substantive item (a metric not yet pinned, a recipe that depends on an rfgen feature not yet shipped), name the deferral explicitly so it is not silent.

5. Context as resource¶

The context window is finite and performance degrades as it fills. Manage it. Use subagents to extend reach without polluting the main thread, and verify their output.

Examples in this repo:

Brief subagents fully. They cannot see the main conversation. Hand over goal, context, what has been ruled out, and the form of the answer.
Verify subagent output. A report describes what an agent intended, not necessarily what it produced. Spot-check the artifact.
Use fresh-context subagents for adversarial review. A reviewer with no exposure to your reasoning catches biases the main thread cannot.

Documentation principles¶

These rules apply to every page under docs/. They flow from the engineering principles above.

EMMA consumes rfgen; it does not generate¶

EMMA is a benchmark library. A benchmark is a dataset plus correct documentation plus an evaluation protocol. EMMA owns dataset recipes (pinned rfgen configurations), loaders, the evaluation harness, the leaderboard, and governance. EMMA does not own emitters, channels, propagation, antenna patterns, or any signal generation. Those live in rfgen. (See the smell test in Part 2.)

A dataset page therefore pins the exact rfgen configuration that regenerates the dataset: rfgen commit, emitters, receiver array, channel environment or environments, labeler, sample counts, splits, and seeds. It does not re-specify RF physics.

Use established libraries and sources¶

Use established libraries, standards, datasets, and file formats when they already define the right concept.

Metrics: torchmetrics, scipy, numpy, torch for the math.
Datasets: rfgen for synthetic scenes, public testbeds (Colosseum, POWDER, COSMOS) for real-capture OOD subsets, SigMF and Zarr for storage and interchange.
Benchmarks as precedent: SUPERB, GLUE and SuperGLUE, MLPerf, ImageNet, WILDS, BIG-bench for the evaluation methodology.

Stable contracts at framework boundaries¶

ABCs and Protocols exist for the boundaries a contributor extends: a frozen backbone, a readout head, a task, a real-capture adapter, a leaderboard store. One ABC per concept; concrete-to-concrete inheritance is forbidden.

The current boundary ABC and Protocol list lives in Part 2: Current boundary contracts.

Organize by reader need¶

Documentation follows the standard split used by mature frameworks:

Section	Reader question	Content style
Getting Started	How do I try the benchmark?	Short path, runnable commands, expected outputs.
Concepts	How should I think about the benchmark?	Mental model, boundaries, data flow, tradeoffs.
Datasets	What data does a task run on and how is it made?	Pinned rfgen recipe, splits, domain relevance.
How-To Guides	How do I complete a task?	Step-by-step procedure for one goal.
Reference	What exactly is the contract?	Signatures, schemas, fields, metrics, formats.
API Reference	What are the Python interfaces?	Classes, methods, enums, lifecycle contracts.
Background	Why is it designed this way?	Rationale, landscape, roadmap, open questions.
Validation	Is each layer scientifically valid?	Six-lens reports per build-plan layer.

Do not put every kind of content on every page. A concept page can link to an API reference. A dataset page can link to a metric. They should not duplicate each other.

Concept page shape¶

A concept page explains how one part of the benchmark fits with the rest. Use this shape when it fits:

# Concept Name
## Overview
## Place In The System
## Boundaries
## Data Flow
## Minimal Example
## Design Notes
## References
## See Also

Do not force this shape when another structure is clearer. The goal is comprehension, not template compliance.

Lead with the reader’s mental model¶

A page must orient the reader before it makes design conclusions. The opening paragraph names the problem the concept solves and the mental model the reader should hold; only after that do design decisions, ABC names, or benchmark-specific idioms appear.

Reference pages define exact contracts¶

Reference pages are precise enough to implement against: purpose, fields, method signatures, parameter and return tables, valid values, invariants, error behavior, examples, and contract tests. Signatures alone are not enough. Narrative alone is not enough.

Make claims testable¶

A design claim implies a validation path.

Weak. The OOD protocol measures generalization.

Better. Under leave-one-environment-out, a model that overfits the channel statistics of the training environment scores no better than the per-task chance baseline on the held-out environment, averaged over held-out environments.

Corresponding validation lives in Validation. If a claim cannot be tested, mark it as rationale, hypothesis, or open question.

Stop and ask when uncertain¶

If a smell test fires, pause and surface the question rather than charging ahead. The cost of one clarifying question is below the cost of being wrong silently.

The current smell-test items live in Part 2: Current smell tests.

What “done” looks like¶

A documentation change is ready when:

All renames are swept across docs, schemas, recipes, examples, anchors, and the link map.
Every code span in prose with an API entry links to its anchor (STYLE.md, code-span linking rule).
There are no em-dashes, no LLM-voice phrasings, and no undefined acronyms.
Closed-set strings are StrEnum members in Python; YAML may use the string value.
Every new claim is testable today or marked proposed-contract or open question.
Every load-bearing claim is cited, and unverified citations carry (verify).
bash scripts/docs_build.sh and python3 scripts/docs_lint.py succeed.

The current build commands live in Part 2: Done-checklist concrete steps.

Part 2: Current design contracts¶

These are the project’s concrete choices today. They follow from Part 1 and will change as the design evolves.

What EMMA owns versus what rfgen owns¶

EMMA owns: dataset recipes (pinned rfgen configs), dataset loaders, label extraction from rfgen metadata, metrics, the frozen-backbone and readout-head protocols, the OOD evaluation harness, the leaderboard and submission system, the CLI, and governance tooling.
rfgen owns: emitter generation, channel and propagation modeling, antenna and array geometry, scene composition, on-disk labels, annotations, and storage.
External sources own: real-capture OOD subsets (Colosseum, POWDER, COSMOS) and any future bridge datasets (DeepMIMO for the CSI bridge).

EMMA references rfgen’s config schema by name; it does not redefine emitter or channel types.

Default libraries¶

Data: rfgen (synthetic, primary), Colosseum, POWDER, COSMOS (real-capture OOD).
Metrics and math: torch, torchmetrics, scipy, numpy.
Schemas and validation: Pydantic v2.
Configuration: Hydra-style composition plus Pydantic validation.
Storage and interchange: SigMF, Zarr, Parquet (predictions and manifests).

Current boundary contracts¶

The current set of framework-owned ABCs and Protocols:

FrozenBackbone (Protocol)
ReadoutHead (ABC)
Task (ABC)
Metric (ABC)
Aggregator (ABC)
OODProtocol (ABC)
RealCaptureAdapter (ABC)
LeaderboardStore (ABC)

Current smell tests¶

If any of these is true, pause and surface the question rather than charging ahead:

“I am about to generate I/Q, or implement an emitter, a channel, a propagation model, or an antenna pattern, in this repo.” This belongs in rfgen, not EMMA.
“I am about to define a metric, an OOD axis, or a task that a standard library or an established benchmark already defines.” Reuse first.
“I am about to cite a paper I have not verified.” Mark it (verify) and stop.
“Two pages disagree about a name, a version, or a default, and I will pick one and move on.”
“This concept page is becoming a 200-line dump.”
“I am writing Literal[...] because the enum does not exist yet.”
“I am adding a backwards-compatibility shim because I am renaming something.”
“A dataset recipe references an rfgen emitter or channel that rfgen v0 does not yet ship.” Flag the dependency; do not pretend the recipe is complete.

Done-checklist concrete steps¶

PATH="$PWD/.venv/bin:$PATH" bash scripts/docs_build.sh succeeds.
python3 scripts/docs_lint.py succeeds (run with the venv interpreter if needed).

Script names may change as the project matures.

Principles¶

Part 1: First principles¶

Engineering principles¶

1. Calibrated humility¶

2. Domain over generic¶

3. Encode rules; do not describe them¶

4. Propagate, then close¶

5. Context as resource¶

Documentation principles¶

EMMA consumes rfgen; it does not generate¶

Use established libraries and sources¶

Stable contracts at framework boundaries¶

Organize by reader need¶

Concept page shape¶

Lead with the reader’s mental model¶

Reference pages define exact contracts¶

Make claims testable¶

Stop and ask when uncertain¶

What “done” looks like¶

Part 2: Current design contracts¶

What EMMA owns versus what rfgen owns¶

Default libraries¶

Current boundary contracts¶

Current smell tests¶

Done-checklist concrete steps¶

See Also¶