Submission¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

A submission is the manifest a submitter sends to the EMMA (Electromagnetic Multi-task Model Assessment) leaderboard. It carries one or more prediction bundles, the model provenance that makes the result reproducible, and, for operator submissions, a pre-registration record. The serialized form is the Submission Pydantic model, materialized as JSON. The leaderboard scores it with HoldoutScorer and posts the resulting LeaderboardRow.

At v0.1 submission is results-only via prediction file. The model itself never leaves the submitter’s machine; only its predictions upload, and EMMA scores them against the secret holdout. At v1 the results-only path is replaced by sandboxed code submission: the submitter sends a runner that materializes the holdout and scores in isolation, so the holdout labels never leave the scoring environment. Both modes write the same on-disk manifest shape described here. See Leaderboard for the narrative contract.

Schema fields¶

Field	Type	Purpose
`leaderboard_version`	ReleaseVersion	The leaderboard track the submission targets, for example `V0_1`.
`model_name`	`str`	Display name of the model on the public board.
`submitter`	`str`	Submitter handle or team name.
`model_provenance`	`ModelProvenance`	Structured provenance record (see below): training config hash, code commit, and weights URI.
`prediction_bundles`	`list[`PredictionBundle]`	One bundle per task in the submission, each carrying its Parquet path and its own `prediction_bundle_hash`.
`pre_registration_id`	`str \| None`	Required for operator submissions; links to the PreRegistration record.

The per-bundle SHA-256 (Secure Hash Algorithm, 256-bit) hash is carried on each PredictionBundle in prediction_bundles, not denormalized onto the submission. A reviewer iterates the list to audit individual bundles; the model half of the result triple surfaces on the linked RunManifest, not on the submission.

Model provenance¶

On disk, model_provenance is a structured record so a reviewer can re-derive the model, not just re-score it. The minimum fields are:

Sub-field	Type	Purpose
`training_config_hash`	`str`	Hash of the resolved training configuration, so a cosmetic edit to the config file that does not change the resolved config leaves the hash unchanged.
`code_commit`	`str`	Git SHA (Secure Hash Algorithm digest) of the training code the model was produced with.
`weights_uri`	`str`	URI of the trained checkpoint, for example an S3 or Hugging Face path.

SubmissionValidator rejects a submission whose model_provenance is absent or incomplete; a posted score without provenance is not auditable, and EMMA does not post unauditable scores. For operator (Superpose) submissions, the validator additionally requires a pre_registration_id, enforcing the clinical-trial-style declaration that the architecture preceded holdout unseal. See Neutrality.

Example¶

{
  "leaderboard_version": "V0_1",
  "model_name": "my-fm@v1",
  "submitter": "team-alpha",
  "model_provenance": {
    "training_config_hash": "sha256:9f2c...1e04",
    "code_commit": "b71d4a8",
    "weights_uri": "s3://team-alpha/my-fm/v1.pt"
  },
  "prediction_bundles": [
    {"task_id": "E_LOC_AOA",  "split": "HOLDOUT", "rows": "preds/E-LOC-AOA.parquet",  "prediction_bundle_hash": "sha256:a1b2...c3d4"},
    {"task_id": "E_ID_DRONE", "split": "HOLDOUT", "rows": "preds/E-ID-DRONE.parquet", "prediction_bundle_hash": "sha256:e5f6...7a8b"}
  ],
  "pre_registration_id": null
}

Version trajectory¶

At v0.1 the ResultsOnlyBoard consumes this manifest via ScoringServer. The submitter runs emma eval locally on the public dev split, produces prediction bundles, and submits them for holdout scoring. At v1 the SandboxedCodeBoard accepts a runner instead of pre-computed predictions; the manifest shape stays the same, with prediction_bundles produced inside the sandbox rather than uploaded.

References¶

Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Reproducible scoring with required provenance; the model-provenance contract this manifest enforces.
Yang et al., “SUPERB: Speech processing Universal PERformance Benchmark,” Interspeech 2021, arXiv:2105.01051. Results-only evaluation of a shared backbone; the v0.1 submission mode. (verify)