COVER-Fish/Reference-Update-Benchmark

COVER-Fish Reference-Update Benchmark

This benchmark studies how reference-corpus updates change a frozen recognition system, instantiated on fine-grained fish identification. It publishes the versioned control plane behind COVER-Fish: row-level manifests, gallery states, taxonomy, tensor bindings, transition evidence, protocols and dependency locks.

The benchmark does not duplicate the 83 GB Full Payload Archive. Large source archives and frozen tensors remain in the immutable release at DOI `10.57967/hf/9776`, revision 4e437b6a2bf5f9a12a200bbe3a93411fe713db1f. The lock in dependencies/DEPENDENCIES.tsv binds every required object by path, byte count and complete SHA-256.

Canonical tracks

1. 1.Gallery-state replay — hold encoder, query roster and aggregation rule fixed while comparing R0, R1, R2, R3, R4, +USFWS, +ANGFA and +Commons.
2. 2.Aggregation replay — hold embeddings and gallery membership fixed while comparing species-centroid, instance-max, source-separated or another fully declared aggregation rule.

Both tracks report query-micro and species-macro top-1/top-5/top-20 together with wrong-to-right and right-to-wrong transitions. This release does not create an acquisition leaderboard over the public QT26-QC labels.

What to download

The evaluation roster is independently published as `COVER-Fish/QT26-QC`. Its E0 copy remains a dependency here so the complete historical replay is closed under the same fixed snapshot.

Minimal Level A replay

python tools/download_dependency.py CORE --output dependencies/downloads
tar --zstd -xf dependencies/downloads/coverfish-rev023-core-metadata-index-code-rc2-20260714.tar.zst \
  -C dependencies/core
python examples/minimal_reproduction.py dependencies/core/index

The fixture uses NumPy, frozen query embeddings and frozen species prototypes; it requires no pixels, network access after download, PyTorch or encoder run.

Licence-aware selection

rows-by-licence/ contains row-ID lists for CC0/public-domain, CC BY, CC BY-SA, non-commercial, permission-required and pointer-only records. These lists make the row-level rights ledger operational: after obtaining and extracting a source archive, tools/extract_rows.py can select the requested rows.

The lists are not separately downloadable image packs. In particular, Commons is one 66.6 GB compressed stream transported in nine parts; selecting a Commons subset requires obtaining and extracting that stream first. Row-level terms remain authoritative.

Dependency verification

Anyone can run python tools/check_dependencies.py to compare the dependency lock with the fixed Hub revision. The tool uses LFS content oids when available, fully hashes small non-LFS files, and can fall back to two fixed byte-range hashes without calling them a complete-file SHA-256. It never executes strings from TSV files. No fixed monitoring frequency or continuing verification service is promised.

Citation

• —Use this benchmark's DOI when using gallery states, embeddings, ledgers or replay protocols.
• —Use the QT26-QC DOI when using the query set or reporting benchmark results.
• —Use the COVER-Fish paper when using its method or scientific conclusions.
• —Cite 10.57967/hf/9776 and name the component when directly using S0–S4, D0, E0 or CORE bytes.

If several objects are used, cite the objects actually downloaded or used to produce the result.