Harness

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The evaluator, evaluation run, protocol runner, and prediction writer. These classes sit between data (owned by rfgen and the real-capture adapters) and reporting (the leaderboard). The harness drives the full OOD (out-of-distribution) pipeline: encode with a frozen backbone, fit a readout head per fold, score each fold with the task metric, and reduce to the headline aggregate. See Architecture for the data-flow diagram.

Class index

Class	Role	Purpose
Evaluator	Entry point	Scores one backbone on one task under an OOD protocol
EvalRun	Result	Holds predictions, per-environment metrics, and a run manifest
ProtocolRunner	Driver	Applies an OOD protocol and aggregates fold scores
PredictionWriter	Output	Writes Parquet per task, schema-validated

---

class emma.harness.Evaluator

Scores one frozen backbone on one task under its OOD protocol. The entry point for local evaluation and for the leaderboard scoring path. The same backbone instance feeds every task, enforcing the SUPERB (Speech processing Universal PERformance Benchmark) contract that the score reflects representation quality, not head capacity.

class Evaluator:
    def __init__(
        self,
        backbone: FrozenBackbone,
        task: Task,
        protocol: OODProtocol,
        split: Split = Split.DEV,
    ) -> None: ...

    def run(self) -> EvalRun: ...

Constructor parameters

Parameter	Type	Default	Purpose
backbone	FrozenBackbone	required	The frozen backbone to evaluate
task	Task	required	The task to score
protocol	OODProtocol	required	The OOD protocol to apply
split	Split	DEV	Which split to evaluate on

Methods

run()

Returns. An EvalRun carrying per-fold predictions, per-environment metrics, and a RunManifest.

Notes

• The readout head is re-fit per fold because the training scenes differ per holdout. The backbone is shared across folds.

• Requesting Split.HOLDOUT outside the scoring server path raises HoldoutAccessError.

---

class emma.harness.EvalRun

The result of one evaluation run. Holds the per-fold predictions, per-environment metric values, and the provenance manifest.

@dataclass(frozen=True, slots=True)
class EvalRun:
    predictions: dict[str, torch.Tensor]
    per_environment_metrics: dict[str, float]
    run_manifest: RunManifest

    @property
    def ood_avg(self) -> float: ...

Fields

Field	Type	Purpose
predictions	dict[str, torch.Tensor]	Per-fold predictions, keyed by held-out value
per_environment_metrics	dict[str, float]	Per-environment (or per-unit, per-band) metric values
run_manifest	RunManifest	Provenance for the run

Properties

Property	Type	Purpose
ood_avg	float	The aggregated OOD average for this run

Notes

• ood_avg is computed by applying the Aggregator (passed to the ProtocolRunner that produced the run) to per_environment_metrics. The result is exposed as a derived property; it carries no state beyond the stored fold metrics.

---

class emma.harness.ProtocolRunner

Applies an OODProtocol and aggregates fold scores. Materializes one fold per held-out value, fits the readout head on the training fold, scores the eval fold, and averages. The frozen backbone is shared across folds; only the lightweight readout head is re-fit.

class ProtocolRunner:
    def __init__(
        self,
        backbone: FrozenBackbone,
        task: Task,
        protocol: OODProtocol,
        aggregator: Aggregator,
    ) -> None: ...

    def run(self, dataset: EMMADataset) -> EvalRun: ...

Constructor parameters

Parameter	Type	Default	Purpose
backbone	FrozenBackbone	required	The frozen backbone shared across folds
task	Task	required	The task providing the metric and head
protocol	OODProtocol	required	The fold-construction protocol
aggregator	Aggregator	required	The fold-score reducer; the only concrete implementation at v0.1 is OODAvg, supplied with the release’s frozen anchor pair

Notes

• The aggregator parameter is typed as the ABC so a test or downstream fork can substitute a stub reducer; production paths always pass OODAvg.

Methods

run(dataset)

Parameter	Type	Purpose
dataset	EMMADataset	The dataset to split into folds

Returns. An EvalRun with per-fold predictions and aggregated metrics.

---

class emma.harness.PredictionWriter

Writes predictions to Parquet, one file per task, schema-validated against the task’s prediction contract. The output format matches what SubmissionValidator expects on the leaderboard path.

class PredictionWriter:
    def __init__(self, output_dir: str) -> None: ...

    def write(self, run: EvalRun, task_id: TaskID) -> PredictionBundle: ...

Constructor parameters

Parameter	Type	Default	Purpose
output_dir	str	required	Directory to write Parquet files into

Methods

write(run, task_id)

Parameter	Type	Purpose
run	EvalRun	The run whose predictions are written
task_id	TaskID	The task the predictions belong to

Returns. A PredictionBundle referencing the written Parquet file. Raises PredictionSchemaError if the predictions fail schema validation.

See Also

• Architecture: the seven-stage pipeline these classes implement.

• OOD protocol: the leave-one-X-out mechanics the protocol runner applies.

• Tasks: the backbone, head, and task contracts the evaluator consumes.

• Leaderboard: the submission path that consumes the prediction writer output.