imageomics/fine-grained-challenges

Dataset Card for Fine-Grained Challenges

Fine-Grained Challenges collects focused groups of visually similar animals for testing biological image classifiers. It combines images, taxonomy, provenance, and frozen embeddings from three BioCLIP-family models in one Lance dataset.

Dataset Details

Dataset Description

• —Curated by: Matthew J. Thompson, Imageomics Institute
• —Language(s) (NLP): English metadata and scientific nomenclature
• —Homepage: https://huggingface.co/datasets/imageomics/fine-grained-challenges
• —Repository: https://huggingface.co/datasets/imageomics/fine-grained-challenges
• —Paper: No associated paper

Release v0.1.0 contains three challenge groups:

The Peromyscus group includes 305 camera-trap images resolved only to genus. They were not included in the training data of the listed BioCLIP models and can be used as an open-ended, field-like inspection set. They are not species-level ground truth.

This repository uses pre-1.0 semantic versioning. During the v0.x series, minor releases may add or revise data, schema, embeddings, or scientific interpretation. Patch releases are reserved for documentation and other changes that do not alter the data. Tags identify fixed release states.

Supported Tasks and Leaderboards

The dataset supports:

• —fine-grained image classification within a challenge group;
• —zero-shot classification with vision-language models;
• —few-shot classifiers trained on frozen image embeddings;
• —comparison of embedding spaces across BioCLIP model generations;
• —inspection of predictions on genus-only camera-trap images;
• —testing retrieval and nearest-neighbor methods.

The repository does not define a leaderboard, a mandatory train-test split, or a single metric. A valid evaluation must describe its split, sampling unit, label set, model revision, text prompts if applicable, and metric.

Dataset Structure

The corpus is a single Lance dataset with embedded image bytes:

data/
|-- challenges.lance/
|   |-- data/
|   |-- _indices/
|   |-- _transactions/
|   `-- _versions/
`-- challenges.embedding_models.json

Lance supports column projection and filter pushdown, so metadata can be inspected without retrieving image bytes or embeddings.

Data Instances

Open the current repository state remotely:

import lance

URI = "hf://datasets/imageomics/fine-grained-challenges/data/challenges.lance"
ds = lance.dataset(URI)

print(ds.count_rows())
print(ds.count_rows(filter="challenge_group = 'Peromyscus'"))

peromyscus = ds.scanner(
    filter="challenge_group = 'Peromyscus'",
    columns=["uuid", "species", "source_dataset", "emb_bioclip2p5"],
).to_table()

For a durable analysis, download a tagged release and open it locally:

from huggingface_hub import snapshot_download
import lance

root = snapshot_download(
    repo_id="imageomics/fine-grained-challenges",
    repo_type="dataset",
    revision="v0.1.0",
    allow_patterns="data/challenges.lance/**",
    local_dir="fine-grained-challenges-v0.1.0",
)
ds = lance.dataset(f"{root}/data/challenges.lance")

Decode an image from its stored WebP bytes:

import io
from PIL import Image

row = ds.take([0], columns=["image", "scientific_name"]).to_pylist()[0]
image = Image.open(io.BytesIO(row["image"]))

Data Fields

The embedding columns are indexed with cosine IVF-FLAT indices. uuid and challenge_group have scalar B-tree indices. Model revisions and embedding norm statistics are recorded in data/challenges.embedding_models.json and in the Arrow field metadata.

Data Splits

No fixed train, validation, or test split is provided.

The 6,328 rows derived from TreeOfLife-200M have seen_in_training = ["bioclip-2", "bioclip-2.5"]. This is an image-membership statement for those named model versions, not a guarantee about other models or about related observations.

The 305 LILA camera-trap rows have seen_in_training = [], species = null, and scientific_name = "Peromyscus". They are suitable for prediction inspection, not supervised species-level scoring.

Users should make splits at the observation, sequence, specimen, site, or source level where that information is available. Random image-level splits can leak near-duplicate or related observations across partitions.

Dataset Creation

Curation Rationale

Broad biological benchmarks can conceal failures among close relatives. These challenge groups concentrate on distinctions that are scientifically meaningful and visually difficult. The dataset also places pretrained embeddings beside the images and metadata so that zero-shot and few-shot methods can be compared without repeating expensive image encoding.

Source Data

The species-labeled images were selected from TreeOfLife-200M. The genus-only camera-trap images came from two collections distributed through LILA BC: the Wildlife Conservation Society Camera Traps and the North American Camera Trap Images dataset.

Source composition in v0.1.0 is:

The four rows whose immediate source is unspecified retain usable per-image license and provenance fields. They were not assigned a source that could not be verified.

Data Collection and Processing

TreeOfLife-200M records were grouped by the selected fine-grained challenges. Species were sampled with a fixed random seed up to 100 images per species. Records resolved only to genus were retained in a separate bucket, also capped at 100 during that sampling stage. Images, taxonomic annotations, provenance, and model embeddings were joined by stable image identifiers.

Rows were excluded when the image license was missing, lacked a license URL, was marked all-rights-reserved or other, or prohibited derivative works. This is necessary because the stored WebP representation and model embeddings are derivative outputs. Every retained row has a license name, license URL, and copyright-owner field.

For the camera-trap cohort, records labeled as Peromyscus were identified before taxonomic resolution. The Ohio Small Animals collection was excluded. Candidate records came from the Wildlife Conservation Society Camera Traps and North American Camera Trap Images collections. Taxonomy was resolved with TaxonoPy 0.2.0. Of 432 candidate image records, 431 could be retrieved. Images were resized with Lanczos resampling to a maximum edge of 720 pixels and encoded as lossless WebP with method 6.

MegaDetector v1000 Redwood was used to screen the camera-trap images. A confidence threshold of 0.30 retained 305 images containing an animal detection: 198 from Wildlife Conservation Society Camera Traps and 107 from North American Camera Trap Images. Detection outputs are not species annotations.

All images in the release decode as lossless WebP and have a maximum edge no greater than 720 pixels.

The three embedding columns were recomputed in FP32 with TensorFloat-32 disabled using the preprocessing configuration pinned with each model. The stored values are raw image embeddings and are not L2-normalized:

Normalize these embeddings before cosine classification or cosine-based model comparison.

Who are the source data producers?

The source photographs were made and published by the people and organizations represented in the original TreeOfLife-200M and LILA BC records. These include biodiversity observers, collection personnel, research teams, and automated camera-trap systems. Per-image ownership and license information remains in each row.

Annotations

Annotation process

TreeOfLife-derived taxonomy follows the resolved rank fields available in the TreeOfLife annotation pipeline. Camera-trap candidates were first identified by their source label and then resolved with TaxonoPy. The camera-trap cohort remains at genus because its source metadata does not establish a species label.

seen_in_training records direct image membership in the documented training corpus for the named BioCLIP versions. It does not infer membership from taxonomic similarity.

MegaDetector provided animal presence and bounding-box screening for the camera-trap cohort. Its detections should not be interpreted as taxonomic labels.

Who are the annotators?

Taxonomic and record metadata originate with the contributing source datasets and were standardized through the TreeOfLife and TaxonoPy processing described above. MegaDetector supplied automated animal detections. No new human species-level annotation was added to the LILA camera-trap images.

Personal and Sensitive Information

The release does not intentionally include names of photographed people, coordinates, or other direct personal identifiers. Camera traps can capture people incidentally. No retained camera-trap image had a MegaDetector person detection at the 0.30 screening threshold, but this automated check is not a guarantee that people are absent.

Source URLs can lead to external records containing location or occurrence information. Users should evaluate the sensitivity of those records before redistribution or publication, particularly for vulnerable species.

Considerations for Using the Data

The challenge groups differ in source, scale, taxonomic coverage, and image setting. Report results per group rather than treating all 6,633 rows as one classification problem.

The TreeOfLife-derived images are in the training corpus of BioCLIP 2 and BioCLIP 2.5. Scores on those rows measure performance on this curated challenge, but they are not held-out estimates for those models.

The LILA rows are independent images for the listed model versions, but they lack species labels. Use them to inspect whether predictions are credible, compare methods, or identify examples for expert review. Do not treat model predictions as ground truth.

Common names can be missing or can repeat the most specific scientific name. Prompt-based evaluation should state exactly how missing common names were handled.

Bias, Risks, and Limitations

• —Source sampling is not representative of biological abundance or geographic occurrence.
• —Class counts are capped for portions of the corpus and should not be used as ecological frequency estimates.
• —Taxonomic names can change, and source records can contain identification errors.
• —Images from aggregators can preserve biases and duplicate relationships from their upstream collections.
• —The release does not provide specimen, individual, sequence, site, or duplicate group identifiers uniformly across every source.
• —Camera-trap filtering favors images detected by MegaDetector and can exclude small, obscured, or unusual animals.
• —Camera-trap images are genus-only and cannot measure species accuracy without independent expert annotation.
• —Precomputed embeddings reproduce only the pinned model revisions and preprocessing configurations.

Recommendations

• —Pin a dataset tag and record the ordered UUID hash for reproducible work.
• —Select rows by challenge_group, source, and training exposure before fitting or scoring a model.
• —Normalize the stored embeddings when the method assumes unit vectors.
• —Fit downstream classifiers separately for each model's embedding space.
• —Group related observations before splitting whenever suitable provenance is available.
• —Inspect per-image licenses and attribution requirements before republishing images.
• —Seek expert review before assigning species labels to the LILA camera-trap cohort.
• —Preserve model revision, preprocessing, prompt, split, and metric details with reported results.

Licensing Information

The dataset compilation, including its selection, curation metadata, embeddings, and indices, is released under CC0-1.0. This does not relicense the images.

Each image retains the license recorded in license_name and license_link, and the associated attribution is recorded in copyright_owner. The retained licenses include public-domain terms and licenses that can require attribution, limit commercial use, or require ShareAlike distribution. The LILA camera-trap images use CDLA-Permissive-1.0 in this release. Users are responsible for following the terms attached to each image.

NoDerivatives, all-rights-reserved, ambiguous other, and missing-license-link records were excluded from this release.

Citation

No formal citation is provided for this release. Link to this dataset repository and cite the applicable source datasets when using these data.

TreeOfLife-200M is available at https://huggingface.co/datasets/imageomics/TreeOfLife-200M and has DOI https://doi.org/10.57967/hf/8980. Source URLs and provenance fields identify the other contributing collections at row level.

Acknowledgements

This work was supported by the Imageomics Institute, which is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under Award #2118240 (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

The curators also acknowledge the source-image contributors, biodiversity data publishers, and camera-trap teams represented in the per-row provenance.

Glossary

• —Challenge group: A set of related, visually confusable taxa selected for a focused analysis.
• —Frozen embedding: A numeric representation produced without updating the pretrained encoder.
• —Genus-only: Taxonomy is resolved to genus but not to a supported species label.
• —Seen in training: The exact image is present in the documented training corpus of a named model version.
• —Zero-shot: Classification using model-aligned text or class prototypes without fitting a classifier to labeled examples from the target task.
• —Few-shot: Adaptation using a small number of labeled examples per class.

More Information

• —BioCLIP model family
• —Lance dataset format
• —TaxonoPy
• —MegaDetector
• —Wildlife Conservation Society Camera Traps
• —North American Camera Trap Images

Dataset Card Authors

Matthew J. Thompson

Dataset Card Contact

Use the Discussions tab in this Hugging Face dataset repository.