Data release manifest

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, schema fields, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

A data release manifest is the release-level envelope. It bundles the SceneManifest records for one or more DatasetRecipe instances into a single content-hashed release, pins the rfgen commit that generated every scene, and records which split each scene belongs to. It is the artifact a leaderboard row cites when it claims a score was produced on a specific, re-derivable dataset version.

The serialized form is the DataReleaseManifest Pydantic model, materialized on disk by DataRelease. One release can carry several recipes (for example the four v0.1 scored recipes plus the real-capture subset), each contributing its own scene manifests and split assignments.

Schema fields

Field

Type

Purpose

release_id

str

Canonical release identifier, for example EMMA-v0.1-rc1.

version

ReleaseVersion

The release the manifest belongs to, for example V0_1.

recipes

list[str]

The dataset_id values bundled in this release, each referencing a DatasetRecipe.

scene_manifests

list[SceneManifest]`

The per-scene provenance records, across all bundled recipes.

split_assignments

dict[str, Split ]

Map of scene_id to split (TRAIN, DEV, HOLDOUT). The holdout scene ids are never published with their labels.

rfgen_commit

str

The single frozen rfgen git SHA that generated every synthetic scene in the release.

content_hash

str

Content hash over the manifest and its referenced scene hashes. The reproducibility anchor for the release as a whole.

Note

A release pins one rfgen_commit for all synthetic recipes. If a later recipe requires a newer rfgen commit, it ships as a new release with a new release_id and content_hash; the old release stays re-derivable from its own pin. A bug is fixed by shipping a new hashed release without invalidating provenance.

Example

# data release manifest (proposal)
release_id: EMMA-v0.1-rc1
version: V0_1
recipes: [EMMA-LOC-v0.1, EMMA-DRONE-v0.1, EMMA-FP-v0.1, EMMA-AMC-v0.1, EMMA-REAL-OOD-v0.1]
scene_manifests: [emma-scene-00017a, emma-scene-00018a, ...]
split_assignments:
  emma-scene-00017a: DEV
  emma-scene-00018a: TRAIN
rfgen_commit: a91f3c2          # placeholder until pinned
content_hash: sha256:c1d4...9a02

References

  • Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML 2021, arXiv:2012.07421. The distribution-shift-first precedent for pinning split assignments at the release level.

See Also