HirakoSan/bio-lens

🌿 iNaturalist Bronze Dataset (Research-Grade, Deduplicated)

Description

This dataset contains research-grade observations from iNaturalist, processed through a bronze-layer pipeline that includes: It's a curated dataset for biodiversity analysis based on community sourced observations, contains millions of images thus can be used for biology model training.

The intent of this data is to train specialist vision models capable of identifying species with high accuracy.

• —Research-grade only observations (community-verified)
• —One photo per observation (deduplicated by observation_uuid, keeping the first photo)
• —AVIF encoded images stored as binary in Parquet
• —~5TB total size across over 41,316 shards

Note: This dataset started as all photos, but was tuned to filter downloaded images to only research-grade observations with 1 photo per observation. This prioritizes observation diversity over multiple (often near-identical) photos of the same specimen.

Data Composition

Top Families

File Structure

data/train/
├── 0000.parquet
├── 0001.parquet
├── ...
└── 41316.parquet.parquet

Schema

Note: Only rows with obs_quality_grade == "research" have the image column populated. Other quality grades have NULL images.

Image Format

• —Format: AVIF (AV1 Image File Format)
• —Quality: 75
• —Dimensions: Original (preserved)
• —Average size: ~15-30 KB
• —Metadata: Stripped (EXIF/XMP removed)

Licenses

Photos include licenses as selected by observers: CC0, CC-BY, CC-BY-NC, CC-BY-SA, CC-BY-NC-SA, CC-BY-ND, CC-BY-NC-ND.

⚠️ Non-commercial research use only

Usage

import polars as pl

# Load only research-grade images
df = pl.read_parquet("data/train/*.parquet")
research_df = df.filter(pl.col("obs_quality_grade") == "research")

# Filter by family
beetles = research_df.filter(pl.col("family") == "Carabidae")

# Access image
row = research_df.first()
image_bytes = row['image']

Processing Pipeline

This dataset was created by:

1. 1.Joining photos, observations, and taxonomy
2. 2.Writing all photos (research + casual + needs_id) to bronze parquet
3. 3.Downloading and converting images only for research-grade observations
4. 4.Storing images in the image column (NULL for non-research-grade)
5. 5.Deduplicating to one photo per observation
6. 6.Sharding into ~500MB Parquet files

Citation

@misc{bio-lens-bronze,
  author = {iNaturalist Contributors},
  title = {iNaturalist Bronze Dataset (Research-Grade, Deduplicated)},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/HirakoSan/bio-lens}}
}

Notes

• —The dataset can be further augmented, it would represent over ~12 TB total
• —Only ~142.8M rows have images (research-grade)
• —The remaining rows (casual/needs_id) have NULL image columns and can be filtered out
• —One photo per observation favours observation diversity