Governance validation

Warning

Pre-implementation. This page describes proposed contracts. Behavior is subject to change before code lands.

Scope: the construct, literature, and methodology validation of the ring-fence instruments documented in Governance: the PreRegistration store, the ReScoringLog writer, and the COICharter scaffolding. As an engineering layer, this report carries construct, literature, and methodology depth; the empirical-realism and robustness-boundaries lenses are lighter here. The contract under validation is that the instruments actually enforce neutrality: pre-registration before unseal, third-party re-scoring with published logs, and a firewall between the Superpose model team and EMMA (Electromagnetic Multi-task Model Assessment) ops.

1. Purpose and construct

Neutrality is the property no model vendor can replicate, and it cannot be manufactured retroactively (see Neutrality). The governance layer is the machinery that makes the property auditable rather than asserted. Construct validity asks whether each instrument enforces the neutrality claim it is named for.

Pre-registration before unseal

PreRegistrationStore freezes a PreRegistration record declaring the seed backbone’s architecture and training configuration before the held-out split is unsealed. The construct rule is temporal: a declaration timestamped after unseal is worthless. The instrument enforces this by setting the timestamp server-side at freeze, so the submitter cannot back-date it, and by content-hashing the frozen record so any later edit produces a different hash.

  • Claim. The operator’s seed-backbone architecture is declared before the held-out is unsealed, clinical-trial-style.

  • Proxy. A frozen, append-only PreRegistration record with a server-set timestamp and a published content hash.

  • Omission. At v0.1 pre-registration is required for operator (Superpose) submissions only; external submitters are not required to pre-register. The construct holds for the operator, the party the ring-fence exists to constrain.

Third-party re-scoring with published logs

ReScoringLogWriter records that an external co-steward re-scored a submission on the held-out split. The construct rule is independence: the same HoldoutScorer runs under a different hand, and the result is published. The ReScoringLog entry identifies the submission by submission_id, carries scored_by so the chain of custody is visible, and carries logs_uri so the run is inspectable. The model-side artifact (per-task PredictionBundle hashes) is recovered from the linked RunManifest, not from the LeaderboardRow, mirroring the row/run hash split audited in Leaderboard validation.

  • Claim. Superpose’s own model is re-scored by an external co-steward on every release, and the result is published.

  • Proxy. An append-only ReScoringLog entry per operator release, with a published log artifact.

  • Omission. The co-steward is a single named party at v0.1; the succession plan is an open risk tracked in Leaderboard validation.

The firewall

The COICharter is the versioned conflict-of-interest (COI) charter text that defines who may grade which submission and when abstention is required, separating the Superpose model team from EMMA ops. The construct rule is separation of interest: the team that builds a model cannot be the team that grades it. The charter is content-hashed so a change to the governance rules is itself auditable.

Sharpest construct threats

(Severity: high, architectural) Neutrality cannot be retrofitted. If the operator’s model is ever scored without an independent re-score, no later instrument undoes the conflict. The construct depends on the re-scoring equivalence holding from day one. Cross-link: Leaderboard validation.

(Severity: medium) The operator-as-evaluator failure mode is the cautionary tale the design inverts. A model-builder that owns the dataset, the model, and the benchmark and reports state-of-the-art on its own bench has no neutral check; the PReD-Bench precedent (cited in Literature) is the documented instance. The ring-fence instruments exist precisely to keep EMMA out of that state, but their enforcement is procedural at v0.1, not yet machine-checked end to end.

(Severity: medium) The single external co-steward is a key-person concentration. If the co-steward is compromised or the held-out seed leaks, the held-out is burned. The succession and seed-rotation plan is not yet documented.

2. Mathematical fidelity

The governance layer carries no formulas. The only arithmetic is content hashing. Each frozen PreRegistration carries an explicit hash field, computed by the SHA-256 (Secure Hash Algorithm) construction audited in Content hashing. Each ReScoringLog entry is content-addressed under its logs_uri by the ReScoringLogWriter, which refuses overwrites, so a re-scoring run is pinned to a unique artifact even though the entry does not expose a hash field. The construct property is that the address is a pure function of the record content, so tampering changes the address.

3. Empirical realism

Governance instruments have no data distribution, so empirical realism is not a load-bearing lens here. The relevant property is operational: the v0.1 ring-fence is a named external co-steward who physically holds the test-set seed and re-scores every Superpose submission, with a published log. Whether that operational arrangement holds under load is a procedural question for the steering committee at v1, not a measurement this report can make before code lands.

4. Literature grounding

The governance design follows established neutral-operator precedents and inverts the operator-as-evaluator failure mode.

  • Neutral-operator sequencing. Mattson et al. (MLPerf) and Russakovsky et al. (ImageNet) are the seeded-with-baselines, governance-follows-adoption precedents; MLCommons formalized neutral governance roughly two years after MLPerf launched. The v0.1 operational ring-fence to v1+ independent non-profit transition follows that sequence.

  • Pre-registration. The ICMJE (International Committee of Medical Journal Editors) clinical-trial registration analogy grounds the temporal declaration before holdout unseal.

  • Anti-gaming. Tramer et al. (stealing models via prediction APIs) grounds the residual adversarial-query risk the re-scoring and anti-overfit guards monitor.

  • Operator-as-evaluator cautionary tale. PReD-Bench (Han et al.) is the documented instance where the same group builds the model, the dataset, and the benchmark and reports state-of-the-art on its own bench; the ring-fence exists to keep EMMA out of that state.

(Severity: low) The ICMJE, Tramer, and PReD-Bench entries carry (verify) in the master reading list; none of their claims are presented as fact here until verified. No further literature entries are required for this report.

5. Experimental methodology and planned tests

Tests are written so an implementer can port them directly to tests/validation/governance/ once src/emma/ lands. The governance layer is amenable to audit-log contract tests because the instruments produce append-only, content-hashed records.

Append-only audit-log test - tests/validation/governance/test_append_only.py

  • Falsifiable claim. Frozen PreRegistration records and written ReScoringLog entries are append-only: a correction produces a new record with a new hash, not an edit to the existing one.

  • Failure mode. A freeze or write call mutates an existing record in place, or two distinct records share a hash.

  • Design. Freeze a pre-registration; attempt to edit it; assert the store rejects the edit and that a second freeze produces a different hash. Write a re-scoring log entry; attempt to overwrite it; assert the writer refuses and that the original logs_uri is unchanged.

  • Sample size. One freeze and one write, plus one correction each.

  • Statistical test. Assertion-based; an append-only store must never mutate.

  • Tolerance. Zero tolerance.

  • Gold-standard reference. The invariants on PreRegistrationStore and ReScoringLogWriter in Governance.

Publishable re-scoring log test - tests/validation/governance/test_rescoring_published.py

  • Falsifiable claim. Every operator submission produces a ReScoringLog entry with a resolvable logs_uri and a non-empty scored_by identifying the external co-steward.

  • Failure mode. An operator submission is posted without a matching re-scoring entry, or the entry’s logs_uri does not resolve.

  • Design. For each operator submission on a release, assert a ReScoringLog exists, that scored_by is not the operator, and that logs_uri resolves to a published artifact.

  • Sample size. Every operator submission, every release (exhaustive).

  • Statistical test. Assertion-based.

  • Tolerance. Zero tolerance; an operator submission without an independent re-score fails the build.

  • Gold-standard reference. The re-scoring equivalence test in Leaderboard validation.

Temporal pre-registration test - tests/validation/governance/test_preregistration_timing.py

  • Falsifiable claim. A PreRegistration timestamp precedes the held-out unseal timestamp for its release.

  • Failure mode. A declaration is back-dated, or the operator declares after unseal.

  • Design. Compare the freeze-set timestamp on the operator’s pre-registration to the release’s holdout-unseal timestamp; assert ordering. Reject any client-supplied timestamp.

  • Sample size. One pre-registration per operator release.

  • Statistical test. Assertion-based ordering.

  • Tolerance. Zero tolerance on ordering; the server sets the timestamp.

  • Gold-standard reference. The declared_before_holdout_unseal field on PreRegistration.

Operator-submission pre-registration requirement test - tests/validation/governance/test_operator_preregistration_required.py

  • Falsifiable claim. An operator submission (from Superpose’s model team) without a pre_registration_id is rejected, and an external submission without one is accepted.

  • Failure mode. The validator accepts an operator submission that bypasses pre-registration, breaking the temporal declaration constraint on the operator, or it wrongly requires pre-registration of an external submitter.

  • Design. Construct two submissions on the same release: one operator submission without pre_registration_id, one external submission without it. Assert SubmissionValidator.validate raises SubmissionRejected on the operator submission and returns None on the external one.

  • Sample size. Two constructed submissions.

  • Statistical test. Assertion-based.

  • Tolerance. Zero tolerance.

  • Gold-standard reference. The SubmissionValidator invariant: operator submissions must carry pre_registration_id; external submissions are not required to.

Intentionally not tested here

Distribution-level probes (sample-size-driven stability, bootstrap CIs, sim-to-real sweeps) do not apply to the governance layer: the instruments produce deterministic, append-only records, not stochastic measurements. Those probes live in Leaderboard validation and Datasets validation. The governance report’s empirical-realism lens (section 3) carries the same posture.

6. Robustness boundaries

  • Operator self-grading. (Severity: high) If the operator’s model is ever scored without an independent re-score, neutrality is broken retroactively. Probe: the publishable re-scoring log test in section 5; absence of a matching ReScoringLog fails the build.

  • Operator bypasses pre-registration. (Severity: medium) If an operator submission is accepted without a pre_registration_id, the operator can choose the architecture after holdout unseal, defeating the temporal declaration. Probe: the operator-submission pre-registration requirement test in section 5; an operator submission without pre_registration_id fails the build.

  • Key-person concentration. (Severity: medium) The single external co-steward is a succession risk; if the seed leaks the held-out is burned. Probe: a documented succession and seed-rotation schedule before the board goes live. Cross-link: Leaderboard validation.

  • Procedural enforcement. (Severity: medium) The v0.1 ring-fence is an operational arrangement, not yet machine-checked end to end. Probe: the four contract tests above, ported before public submission opens.

7. Synthesis

Load-bearing claims supported. The instruments are the right construct for neutrality: pre-registration makes the operator’s architecture declaration temporal and tamper-evident, the re-scoring log makes independent scoring published and append-only, and the COI charter makes the firewall versioned and auditable. The design follows the MLPerf and ImageNet neutral-operator sequencing and inverts the PReD-Bench operator-as-evaluator failure mode.

Gaps surfaced. (1) Neutrality cannot be retrofitted, so the re-scoring equivalence must hold from day one; the enforcement is procedural at v0.1, not yet machine-checked. (2) The single external co-steward is a key-person concentration; the succession and seed-rotation plan is not yet documented. (3) Pre-registration is required for operator submissions only, so the construct holds for the constrained party but not for external submitters.

Recommended actions. (1) Land the append-only, publishable re-scoring, temporal pre-registration, and operator-submission pre-registration requirement contract tests as the first tests/validation/governance/ entries before public submission opens. (2) Document the co-steward succession and seed-rotation schedule. (3) Wire the re-scoring equivalence test from Leaderboard validation so an operator submission without a matching independent re-score fails the build.

Top three findings.

  1. (high) Neutrality cannot be retrofitted; the operator’s model must be independently re-scored on every release from day one, and the publishable re-scoring log test is the proposed first verification.

  2. (medium) The single external co-steward is a key-person concentration; the succession and seed-rotation schedule must be documented before the board goes live.

  3. (medium) The v0.1 ring-fence is procedurally enforced, not yet machine-checked end to end; the four contract tests close the residual risk before public submission.

References

  • Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Neutral governance formalized after launch; the sequencing precedent for the v1+ independent-operator transition.

  • Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” IJCV 2015, arXiv:1409.0575. Seeded with baselines; governance followed adoption.

  • International Committee of Medical Journal Editors (ICMJE), “Clinical Trial Registration.” The pre-registration analogy for the operator seed-backbone declaration. (verify)

  • Tramer, Zhang, Juels, Reiter, and Ristenpart, “Stealing Machine Learning Models via Prediction APIs,” USENIX Security Symposium 2016, arXiv:1609.02943. The adversarial query-extraction residual risk the re-scoring and anti-overfit guards monitor. (verify)

  • Han et al., “PReD: Pretrained Remote-sensing foundation model” and PReD-Bench, 2026, arXiv:2603.28183. The operator-as-evaluator cautionary tale the ring-fence inverts. (verify)

See Also