thompsonmj/test-fine-grained-challenges

Fine-Grained Challenges

Groups of morphologically similar species for evaluating and tuning fine-grained classifiers on top of BioCLIP ecosystem model embeddings. Each group gathers species that are easily confused with one another, and the groups span several clades so a classifier can be probed on the distinctions that actually matter rather than on coarse taxonomy.

The corpus lives in a Lance dataset and can be acted on as a whole, on any single group independently, or on any single image. Images, embeddings from three BioCLIP model generations, taxonomic lineage, provenance, and search indexes all live in the same table:

data/challenges.lance/

Schema

seen_in_training is per-image and is constant within a source cohort, so a single group can mix images seen in training with held-out images. See Considerations for using this data.

Embedding columns contain raw, unnormalized encoder output stored as float32. Normalize downstream when using cosine similarity or when an analysis requires unit-length features. Model repository IDs, revisions, dimensions, preprocessing, and software versions are recorded in data/challenges.embedding_models.json and in each embedding field's Arrow metadata.

Groups

Each group is a set of confusable species drawn from TreeOfLife-200M. Images are sampled at up to 100 per species with a fixed random seed, so classes stay roughly balanced; images that resolve only to a genus are kept as a separate bucket, also capped at 100.

The species-labeled groups are drawn from TreeOfLife-200M. The Peromyscus group also includes 305 genus-only camera-trap images from WCS Camera Traps and NACTI. MegaDetector v1000 Redwood retained frames with an animal detection confidence of at least 0.30. These camera-trap rows have null species values and seen_in_training = []; they are intended for open-ended prediction and inspection, not species-level accuracy measurement.

Usage

Install: uv pip install pylance polars pillow

import lance
import polars as pl
URI = "hf://datasets/thompsonmj/test-fine-grained-challenges/data/challenges.lance"
ds = lance.dataset(URI) # scans remotely over hf://, no full download

# Groups present and their sizes
pl.from_arrow(ds.to_table(columns=["challenge_group"]))["challenge_group"].value_counts()

# One group independently (filter pushed down to only retrieve relevant rows and column projection to only retrieve relevant columns)
hummingbirds = ds.scanner(
    filter="challenge_group = 'Trochilidae'",
    columns=["uuid", "species", "emb_bioclip2p5"],
).to_table()

# Select by training exposure (unseen test set vs. seen in training)
unseen = ds.scanner(filter="array_length(seen_in_training) = 0").to_table()
seen_by_bioclip2 = ds.scanner(
    filter="array_has(seen_in_training, 'bioclip-2')").to_table()

# Image bytes (raw webp), same pattern as the HF Lance image example
import io
from PIL import Image
row = ds.take([0], columns=["image", "species"]).to_pylist()[0]
img = Image.open(io.BytesIO(row["image"]))

# Materialize (i.e. download) a group locally for heavy/training use (avoids Hub rate limits)
lance.write_dataset(hummingbirds, "./Trochilidae.lance")

Because reads are columnar and pushed down over hf://, you can browse metadata cheaply by projecting to just the columns you need (e.g. omit image and the emb_* columns). For example, count the images of one species without fetching any images or embeddings:

ds.count_rows(filter="species = 'Peromyscus maniculatus'")

Considerations for using this data

The seen_in_training field records the model versions whose training data included each image, so in-distribution evaluation can be separated from unseen tests:

For TreeOfLife-200M rows, seen_in_training is derived per image from its presence in the training data for the listed model versions. The LILA camera-trap images were sourced separately and are marked unseen. They have only a genus label, so use them for prediction inspection rather than species-level evaluation.

Models referenced: `bioclip`, `bioclip-2`, and `bioclip-2.5-vith14`. The corresponding columns are emb_bioclip, emb_bioclip2, and emb_bioclip2p5.

Licensing and attribution

The dataset compilation (curation, metadata, embeddings, and indexes) is released under CC0-1.0, matching imageomics/TreeOfLife-200M. This does not relicense the images.

Each image keeps its own license, recorded in its row (license_name, license_link, copyright_owner) from TreeOfLife-200M's provenance.parquet.

No-derivatives (*-nd) licenses are excluded at build time, since the stored webp and embeddings are derivative works. Images whose license cannot be determined from provenance are excluded for the same reason. The remaining mix is mostly NonCommercial (*-nc-*), with some ShareAlike (*-sa-*), plain attribution, and public-domain images.

Use each image under its own license.

Per-license counts, computed from the data (URI as defined above):

import lance
import polars as pl
t = lance.dataset(URI).to_table(columns=["challenge_group", "license_name"])
pl.from_arrow(t).group_by(["challenge_group", "license_name"]).len()  # license counts, per group