Add a task

Note

The contracts on this page are proposed until the CLI (command-line interface) and task registry ship. The contribution mechanism follows the BIG-bench (Beyond the Imitation Game benchmark) gated-curation model: a community owns a task, the steering committee curates it.

Contribute a new task to EMMA (Electromagnetic Multi-task Model Assessment). Every task runs on the same frozen backbone and the same raw multi-antenna I/Q (in-phase and quadrature) substrate, so adding a task means defining what to measure, not building new infrastructure. A task is a TaskSpec, a ReadoutHead, a Metric, an OOD (out-of-distribution) axis, and a dataset recipe. Curation is gated by the steering committee to keep the aggregate score defensible.

Goal

Define a task end to end and submit it for committee review, so that it ships in a release as a scored column on the leaderboard.

Prerequisites

  • You have read Task reference and confirmed no existing task already measures your capability.

  • You have a target domain and a real community that depends on the capability. A task with no downstream buyer is a finding, not a feature (see Tasks validation).

  • You have a reproducible rfgen recipe for the scenes, or a plan for one.

Steps

  1. Write a TaskSpec. The spec wires a TaskID to its metric, its OOD axis, its readout head class, and its target Domain. The spec is the contract the rest of the harness reads; everything else is an implementation of it.

  2. Pick the metric. Reuse a standard-library metric before defining a new one: torchmetrics, scipy, or numpy. Only define a new Metric subclass when no established library covers the quantity you measure (see Metrics reference).

  3. Pick the OOD axis. The axis is the contract: a task without one is not fully specified. Use leave-one-environment-out, leave-one-unit-out, or leave-one-band-out where they fit; only propose a new axis after searching the established precedents (see OOD protocol).

  4. Implement the readout head. Subclass ReadoutHead so the head consumes frozen-backbone representations and produces task outputs. The head is lightweight; the backbone carries the representational load (see Add a readout head).

  5. Pin the dataset recipe. Write a DatasetRecipe that freezes the rfgen commit, resolved configuration, seed range, antenna geometry, and label-extraction map. If the recipe depends on an rfgen feature not yet shipped, flag the dependency; do not pretend the recipe is complete.

  6. Submit to the steering committee. Open a task proposal with the spec, the head, the metric, the axis, and the recipe. The committee reviews for construct validity, domain utility, and overlap with existing tasks. Accepted tasks ship in the next release with a version badge and a maturity label. See Neutrality for the ring-fence that keeps curation independent of any model team.

Expected result

An accepted task lands in the registry, appears in emma list-tasks, and joins the leaderboard at its shipped version. The contributing community owns the task’s spec and recipe across releases.

See Also

  • Task spec: the schema a proposal centers on.

  • Tasks: the readout-head and OOD-protocol API surfaces a task implements.

  • Neutrality: the steering committee and gated curation.

  • Tasks validation: the construct-validity bar a new task must clear.

  • Add a readout head: the head-subclass step in depth.