Anti-overfit¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

The anti-overfit guard is how the private held-out resists overfitting and deliberate gaming. A held-out test set loses its value the moment a submitter can train against it, probe it, or re-roll submissions until a lucky run lands on top. EMMA (Electromagnetic Multi-task Model Assessment) cannot make that impossible by fiat; it makes it detectable and expensive. Four instruments combine: submission rate limiting, query auditing, the frozen-secret-seed property, and pre-registration of the operator’s seed backbone before the held-out unseals. The guard is implemented by AntiOverfit, invoked by ScoringServer on every submission, and it runs against scores produced by HoldoutScorer. The narrative motivation lives in Neutrality; the content-hash mechanism that makes the held-out a secret seed lives in Content hashing.

The four instruments¶

Submission rate limiting¶

AntiOverfit enforces a cap on submissions per submitter per window (default five per day, configurable). The cap raises the cost of the simplest gaming strategy: submitting many variants and keeping the best. With a two-environment held-out at v0.1 (see OOD protocol), the aggregate has high variance, so unrestricted re-submission would let noise rather than quality determine rank. The rate limit turns that noise into an expensive signal. A submission that exceeds the cap is rejected with a rate_limit reason by AntiOverfit.check.

Query auditing¶

The guard audits the sequence of predictions a submitter sends. Repeated near-duplicate predictions, small perturbations of a prior bundle, or predictions whose per-scene values cluster too tightly around a probing target are patterns consistent with extracting information about the held-out labels rather than evaluating a model. When AntiOverfit flags a query pattern, it rejects the submission with a context reason and records the flag in the audit log. The audit does not assert malice; it refuses to score a submission whose information content is dominated by probe attempts.

Frozen-secret-seed property¶

The held-out split is never released as a file. It is a frozen secret seed range held by the external co-steward, re-materialized only inside the scoring path (see Content hashing). Because the data never exists on disk for anyone but the co-steward, the generator cannot become an overfitting oracle: a submitter cannot download the held-out and train on it, the failure mode that ended several static-file benchmarks. A bug in the held-out is fixed by re-materializing under a new secret seed and shipping a new hashed release; the old release remains auditable but superseded.

Pre-registration before unseal¶

The operator’s (Superpose’s) own seed backbone is pre-registered before the held-out is unsealed. The architecture and training configuration are declared, timestamped, and content-hashed in a PreRegistration record, clinical-trial-style, so a model-builder cannot retroactively claim an architecture choice after inspecting the held-out. Operator submissions must carry a pre_registration_id that links to that record, enforced by SubmissionValidator. This is the ring-fence instrument between Superpose’s model team and EMMA ops; it is described in Neutrality and audited by the external co-steward’s re-scoring, recorded in a ReScoringLog.

How the instruments combine¶

submission ──► SubmissionValidator ──► AntiOverfit.check ──► HoldoutScorer.score ──► LeaderboardRow
                   (schema,             (rate limit,           (secret-seed         (audit hashes)
                    provenance,          query audit)           holdout)
                    pre-registration)

The four instruments target four distinct gaming strategies, and each closes a gap the others leave open. Rate limiting closes brute-force re-submission. Query auditing closes information extraction through repeated queries. The frozen-secret-seed closes training on a leaked file. Pre-registration closes the operator retroactively fitting its own model to the held-out. None is sufficient alone; together they make the held-out expensive to probe and impossible to train on directly.

Residual risk: adversarial query extraction¶

(Severity: medium, monitored) The instruments do not make adversarial extraction impossible. A submitter with enough submissions and carefully chosen predictions can in principle extract partial information about held-out labels from the score feedback itself, the attack documented by Tramer et al. for model stealing via prediction APIs. EMMA’s posture is calibrated humility: the rate limit bounds the number of queries, the query auditor flags the patterns the attack produces, and the held-out is re-materialized under a new secret seed on a fixed schedule so any partial extraction ages out. The residual risk is named explicitly, monitored through the audit log, and the subject of a proposed probe test (see Leaderboard validation).

References¶

Tramer, Zhang, Juels, Reiter, and Ristenpart, “Stealing Machine Learning Models via Prediction APIs,” USENIX Security Symposium 2016, arXiv:1609.02943. The adversarial query-extraction attack the residual risk names and the query auditor monitors. (verify)
International Committee of Medical Journal Editors (ICMJE), “Clinical Trial Registration.” The pre-registration analogy for the operator seed-backbone declaration. (verify)
Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. A held-out under controlled shift as a generalization contract; the frozen-secret-seed property extends it.
Mattson et al., “MLPerf Training Benchmark,” MLSys 2020, arXiv:1910.01500. Submission rules and auditable scoring at benchmark scale; the rate-limit and audit-log precedent.