Leaderboard

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

Leaderboard stores, holdout scoring, submission validation, and anti-overfit. These classes implement the reporting surface and the ring-fence that protects it. The leaderboard never shows a raw per-fold number as the headline; OODAvg aggregates first. See Leaderboard for the narrative contract.

Three roles

Role	Examples
ABC	LeaderboardStore
Concrete stores	ResultsOnlyBoard (v0.1), SandboxedCodeBoard (v1)
Scoring and governance	HoldoutScorer, ScoringServer, AntiOverfit, SubmissionValidator

Class index

Class	Role	Version	Purpose
LeaderboardStore	ABC	all	Post, get, and list leaderboard rows
ResultsOnlyBoard	Store	v0.1	Prediction-submission store
SandboxedCodeBoard	Store	v1	Sandboxed-code-submission store
HoldoutScorer	Scorer	v0.1	Scores predictions against the secret holdout; never releases labels
ScoringServer	Server	v0.1	Prediction endpoint at v0.1; sandboxed runner at v1
AntiOverfit	Guard	v0.1	Submission rate-limiting and query auditing
SubmissionValidator	Validator	v0.1	Validates a submission before scoring

---

class emma.leaderboard.LeaderboardStore(ABC)

The leaderboard storage contract. One ABC per concept; concrete stores differ by submission mode, not by storage backend.

class LeaderboardStore(ABC):
    @abstractmethod
    def post(self, row: LeaderboardRow) -> None: ...

    @abstractmethod
    def get(self, version: ReleaseVersion) -> list[LeaderboardRow]: ...

    @abstractmethod
    def list(self, version: ReleaseVersion) -> list[LeaderboardRow]: ...

Abstract methods

Method	Returns	Purpose
post(row)	None	Record a scored row
get(version)	list[{ref}LeaderboardRow ]	Fetch all rows for a version (alias for list)
list(version)	list[{ref}LeaderboardRow ]	List all rows for a version

Notes

• get and list are intentionally identical at v0.1; both return the full row set for a version. The two-method surface preserves room for a paginated or filtered get to diverge at v1 without breaking list callers.

• Rank assignment is the store’s responsibility because HoldoutScorer cannot know other rows when it scores one submission. A store implementation computes LeaderboardRow.rank from ood_avg (higher is better) across the version, either on post or as a derived view on list. The concrete policy (re-rank on every post versus rank-on-read) is left to the store implementation at v0.1.

---

class emma.leaderboard.ResultsOnlyBoard(LeaderboardStore)

The v0.1 leaderboard store. Accepts prediction submissions: the submitter sends a PredictionBundle, EMMA scores it on the secret holdout, and the resulting LeaderboardRow is posted.

class ResultsOnlyBoard(LeaderboardStore):
    def __init__(self, store_uri: str) -> None: ...

    def post(self, row: LeaderboardRow) -> None: ...
    def get(self, version: ReleaseVersion) -> list[LeaderboardRow]: ...
    def list(self, version: ReleaseVersion) -> list[LeaderboardRow]: ...

Parameter	Type	Default	Purpose
store_uri	str	required	URI of the backing storage

---

class emma.leaderboard.SandboxedCodeBoard(LeaderboardStore)

The v1 leaderboard store. Accepts sandboxed-code submissions: the submitter sends a runner that materializes the holdout and scores in isolation, so the holdout labels never leave the scoring environment.

class SandboxedCodeBoard(LeaderboardStore):
    def __init__(self, store_uri: str, sandbox_image: str) -> None: ...

    def post(self, row: LeaderboardRow) -> None: ...
    def get(self, version: ReleaseVersion) -> list[LeaderboardRow]: ...
    def list(self, version: ReleaseVersion) -> list[LeaderboardRow]: ...

Parameter	Type	Default	Purpose
store_uri	str	required	URI of the backing storage
sandbox_image	str	required	Container image used for sandboxed execution

Notes

• The sandbox image is the trust boundary: submitted code runs inside it with no network egress and read-only access to the holdout scenes materialized by DataRelease. The scoring result is the only artifact that leaves the sandbox.

• v0.1 does not ship this store; it goes live with SubmissionMode.SANDBOXED_CODE at v1.

---

class emma.leaderboard.HoldoutScorer

Scores predictions against the secret holdout split. The holdout labels never leave the scoring path; only the resulting metric values are returned. Used by both the v0.1 prediction endpoint and the v1 sandboxed runner.

class HoldoutScorer:
    def __init__(
        self,
        release: DataRelease,
        tasks: list[Task],
        aggregator: OODAvg,
    ) -> None: ...

    def score(self, submission: Submission) -> LeaderboardRow: ...

Constructor parameters

Parameter	Type	Default	Purpose
release	DataRelease	required	The data release providing the holdout scenes
tasks	list[{ref}Task ]	required	The tasks to score
aggregator	OODAvg	required	The aggregator with frozen anchors

Methods

score(submission)

Parameter	Type	Purpose
submission	Submission	The validated submission to score

Returns. A LeaderboardRow with per-task scores, OOD average, and sim-to-real gap.

Invariants

• The holdout labels are never serialized outside the scoring path.

• Superpose’s own model is re-scored by an external co-steward on every release, recorded in a ReScoringLog.

---

class emma.leaderboard.ScoringServer

The submission endpoint. At v0.1 it accepts prediction bundles and scores them via HoldoutScorer. At v1 it accepts sandboxed code runners.

class ScoringServer:
    def __init__(
        self,
        store: LeaderboardStore,
        scorer: HoldoutScorer,
        validator: SubmissionValidator,
        anti_overfit: AntiOverfit,
    ) -> None: ...

    def submit(self, submission: Submission) -> LeaderboardRow: ...

Constructor parameters

Parameter	Type	Default	Purpose
store	LeaderboardStore	required	The leaderboard backing store
scorer	HoldoutScorer	required	The holdout scorer
validator	SubmissionValidator	required	The submission validator
anti_overfit	AntiOverfit	required	The rate-limit and audit guard

Methods

submit(submission)

Validates, scores, and posts a submission. Returns the resulting LeaderboardRow. Raises SubmissionRejected on validation or rate-limit failure.

---

class emma.leaderboard.AntiOverfit

Guards against leaderboard gaming. Enforces submission rate limits and audits scoring queries for patterns consistent with probing the holdout.

class AntiOverfit:
    def __init__(
        self,
        max_submissions_per_day: int = 5,
        audit_log_uri: str | None = None,
    ) -> None: ...

    def check(self, submission: Submission) -> None: ...

Constructor parameters

Parameter	Type	Default	Purpose
max_submissions_per_day	int	5	Rate limit per submitter per day
audit_log_uri	str | None	None	URI for the audit log

Methods

check(submission)

Returns None if the submission passes the rate-limit and audit checks. Raises SubmissionRejected with a context reason if the rate limit is exceeded or the query pattern is flagged.

---

class emma.leaderboard.SubmissionValidator

Validates a submission before scoring. Checks that each PredictionBundle has the correct schema for its task, that the model provenance is present, and that a PreRegistration is attached when required (for operator submissions).

class SubmissionValidator:
    def __init__(self, release: DataRelease) -> None: ...

    def validate(self, submission: Submission) -> None: ...

Constructor parameters

Parameter	Type	Default	Purpose
release	DataRelease	required	The release the submission targets

Methods

validate(submission)

Returns None if the submission is valid. Raises PredictionSchemaError for schema violations or SubmissionRejected for provenance gaps.

See Also

• Leaderboard: the narrative version of the reporting contract.

• Governance: the ring-fence instruments (pre-registration, re-scoring log).

• Anti-overfit: the anti-gaming algorithm reference.

• Reproducibility: the auditable result triple each row carries.