Prediction bundle

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

A prediction bundle is the on-disk prediction format a submitter sends to the leaderboard. It is one Parquet file per task, keyed by scene_id, with columns that differ by task family. The bundle is the unit the scoring path consumes: the model itself never leaves the submitter’s machine at v0.1, only its outputs do. The serialized form is the PredictionBundle data type, written by PredictionWriter on the local evaluation path and re-validated by SubmissionValidator on the leaderboard path before any score is computed.

The bundle carries a prediction_bundle_hash, a SHA-256 (Secure Hash Algorithm, 256-bit) digest over the Parquet bytes. That hash is the model half of the auditable result triple: it pins exactly which outputs were scored, so a posted score is tied to a bit-identical artifact and cannot drift. The hash surfaces for audit on the linked RunManifest (one per task), not on the LeaderboardRow; a reviewer recovers it from the run record. See Content hashing.

Schema fields

Field	Type	Purpose
task_id	TaskID	The task the predictions are for, for example E_LOC_AOA.
split	Split	The split the predictions cover, typically HOLDOUT for a submission.
rows	str	Path to the Parquet file containing the prediction rows.
prediction_bundle_hash	str	SHA-256 content hash of the Parquet file; pins exactly which outputs were scored.

Note

The prediction_bundle_hash is computed over the canonical Parquet bytes written by PredictionWriter, not over an in-memory tensor. A re-score of the same bundle file yields the same hash and the same metric values. The data half of the result triple is the data-release hash on the LeaderboardRow; the metric half is the published scoring code and its frozen anchors.

Per-task-family columns

One Parquet file per task. The scene_id column is mandatory for every family and must match the scene identifiers in the SceneManifest records of the targeted release. Prediction columns differ by family.

Task family	Tasks	Key column	Prediction columns
Angular regression	E-LOC-AOA	scene_id	azimuth_deg, elevation_deg (elevation at v1)
Classification / detection	E-LOC-LOS, E-ID-DRONE, E-ID-FP, E-ID-AMC	scene_id	One probability column per class, for example p_los, p_nlos
Beam management	E-CH-BEAM	scene_id	beam_rank (ordered list of beam indices, or top-k columns)
Vector regression / CSI	E-CH-CSI	scene_id	The predicted channel vector, stored as a fixed-width list column matching the scene’s array geometry
Captioning / QA	E-S2T-CAP, E-S2T-QA	scene_id	The generated text string (caption or answer span)

EMMA reads the array geometry into ArraySpec at load time, so the vector-regression column width is determined by the recipe, not chosen by the submitter. CSI (channel state information) is the v1 bridge task; its column layout finalizes with the recipe.

Example Parquet schema

The angular-regression family (E-LOC-AOA) is the v0.1 flagship. Its Parquet schema is:

scene_id:        string       # matches SceneManifest.scene_id
azimuth_deg:     float32      # predicted angle, degrees; circular distance scored
elevation_deg:   float32      # predicted elevation; lands at v1, optional at v0.1

A classification family (E-ID-DRONE) carries one probability column per class:

scene_id:        string
p_background:    float32      # calibrated probability per class
p_drone_model_a: float32
p_drone_model_b: float32
...

Validation

SubmissionValidator checks, per bundle, that the Parquet schema matches the task’s prediction contract before scoring. The check rejects: a missing or duplicate scene_id; a wrong column set for the task family; a probability vector that does not sum to one where the contract requires it; or a prediction count that does not match the targeted release’s holdout scene count. A schema violation raises PredictionSchemaError and no score is computed. The same schema contract is enforced locally by PredictionWriter, so a bundle that passes local evaluation passes submission validation.

References

• Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Auditable, reproducible scoring at benchmark scale; the prediction-artifact-as-evidence precedent the bundle hash supports.

• Apache Parquet, “Parquet File Format.” Columnar on-disk format for prediction rows. (verify)

See Also

• Submission: the manifest that carries one or more bundles.

• Run manifest: where the bundle hash surfaces as the model half of the result triple.

• Leaderboard row: the posted row whose data half the bundle hash pairs with.

• Content hashing: how the bundle hash is computed and audited.

• Harness: the PredictionWriter API surface.