imageomics/VLM4Bio

1---2license: cc-by-4.03task_categories:4- visual-question-answering5- zero-shot-image-classification6- zero-shot-object-detection7tags:8- biology9- organism10- fish11- bird12- butterfly13- image classification14- zero-shot15- traits16- trait-detection17- vlms18- benchmarks19- CV20- trait identification21- trait grounding22- trait referring23- trait counting24- species classification25language:26- en27pretty_name: VLM4Bio28size_categories:29- 10K<n<100K30configs:31- config_name: Fish32  data_files:33  - split: species_classification34    path: datasets/Fish/metadata/metadata_10k.csv35  - split: species_classification_easy36    path: datasets/Fish/metadata/metadata_easy.csv37  - split: species_classification_medium38    path: datasets/Fish/metadata/metadata_medium.csv39  - split: species_classification_prompting40    path: datasets/Fish/metadata/metadata_prompting.csv41- config_name: Bird42  data_files:43  - split: species_classification44    path: datasets/Bird/metadata/metadata_10k.csv45  - split: species_classification_easy46    path: datasets/Bird/metadata/metadata_easy.csv47  - split: species_classification_medium48    path: datasets/Bird/metadata/metadata_medium.csv49  - split: species_classification_prompting50    path: datasets/Bird/metadata/metadata_prompting.csv51- config_name: Butterfly52  data_files:53  - split: species_classification54    path: datasets/Butterfly/metadata/metadata_10k.csv55  - split: species_classification_easy56    path: datasets/Butterfly/metadata/metadata_easy.csv57  - split: species_classification_medium58    path: datasets/Butterfly/metadata/metadata_medium.csv59  - split: species_classification_hard60    path: datasets/Butterfly/metadata/metadata_hard.csv61  - split: species_classification_prompting62    path: datasets/Butterfly/metadata/metadata_prompting.csv63description: "Benchmark dataset of scientific question-answer pairs used to evaluate pretrained VLMs for trait discovery from biological images. VLM4Bio consists of images of three taxonomic groups of organisms: fish, birds, and butterflies, each containing around 10k images. Scientific tasks supported by this dataset include species classification and trait identification, referring, grounding, and counting."64---65 66# Dataset Card for VLM4Bio67 68## **Instructions for downloading the dataset**69 70* Install [Git LFS](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage)71* Git clone the VLM4Bio repository to download all metadata and associated files72  * Run the following commands in a **terminal**:73```bash74git clone https://huggingface.co/datasets/imageomics/VLM4Bio75cd VLM4Bio76```77 78**Downloading and processing bird images**79* To download the bird images, run the following command:80```bash81bash download_bird_images.sh82```83* This should download the bird images inside `datasets/Bird/images`84 85**Processing fish and butterfly images**:86* Run the following command:87```bash 88bash process_fish_butterfly_images.sh89```90* This should process fish and butterfly images inside `datasets/Fish/images` and `datasets/Butterfly/images` respectively. 91 92![Alt text](assests/teaser.png)93 94## Dataset Details95VLM4Bio is a benchmark dataset of scientific question-answer pairs used to evaluate pretrained VLMs for trait discovery from biological images. VLM4Bio consists of images of three taxonomic groups of organisms: fish, birds, and butterflies, each containing around 10k images. 96 97- **Repository:** [VLM4Bio GitHub](https://github.com/Imageomics/VLM4Bio)98- **Paper:** [arXiv](https://arxiv.org/pdf/2408.16176)99 100### Dataset Description101VLM4Bio is a large, annotated dataset, consisting of 469K question-answer pairs involving around 30K images from three groups of organisms: fish, birds, and butterflies, covering five biologically relevant tasks.102The scientifically relevant tasks in organismal biology includes species classification, trait identification, trait grounding, trait referring, and trait counting.103These tasks are designed to test different facets of VLM performance in organismal biology, ranging from measuring predictive accuracy to assessing their ability to reason about their predictions using visual cues of known biological traits.104For example, the tasks of species classification test the ability of VLMs to discriminate between species, while in trait grounding and referring, we specifically test if VLMs are able to localize morphological traits (e.g., the presence of fins of fish or patterns and colors of birds) within the image.105We consider two types of questions in this dataset. First, we consider open-ended questions, where we do not provide any answer choices (or options) to the VLM in the input prompt.106The second type is multiple-choice (MC) questions, where we provide four choices of candidate answers for the VLM to choose from (out of which only one is correct while the remaining three are randomly selected from the set of all possible answers).107 108### Supported Tasks and Leaderboards109The following figure illustrates VLM4Bio tasks with different question types.110![Alt text](assests/tasks.png)111 112The following table demonstrates the leaderboard of the VLM baselines in terms of zero-shot accuracy.113![Alt text](assests/leaderboard.png)114 115### Languages116English, Latin117 118## Dataset Structure119After downloading and processing the dataset, VLM4Bio should have the following structure:120 121```122VLM4Bio/123    └── datasets/124        ├── Fish/125        │   ├── images/126        │   │   ├── INHS_FISH_58870.jpg127        │   │   ├── INHS_FISH_58819.jpg128        │   │   └── ...129        │   └── metadata/130        │       ├── metadata_10k.csv131        │       ├── metadata_500.csv132        │       └── ...133        ├── Bird/134        │   ├── images/135        │   │   ├── Ivory_Gull_0117_49227.jpg136        │   │   ├── Yellow_Warbler_0026_176337.jpg137        │   │   └── ...138        │   └── metadata/139        │       ├── bird_metadata_10k.csv140        │       ├── identification.csv141        │       └── ...142        └── Butterfly/143            ├── images/144            │   ├── butterfly_train_heliconius_sara_0007.jpg145            │   ├── butterfly_val_pyrrhogyra_cramen_0001.jpg146            │   └── ...147            └── metadata/148                ├── metadata.csv149                └── imagelist.csv150```151 152### Data Instances153<!--154Describe data files155 156Ex: All images are named <img_id>.png, each within a folder named for the species. They are 1024 x 1024, and the color has been standardized using <link to color standardization package>.157-->158 159 160### Data Fields161 162**Fish Files:**163- `identification_imagelist_10k.txt`:164- `identification_matrix.csv`:165- `imagelist_10k.txt`: List of image filenames for species classification and trait identification.166- `imagelist_500.txt`: 500 image subset of `imagelist_10k.txt` for trait detection and counting.167- `metadata_10k.csv`: Image filenames (`fileNameAsDelivered`, unique identifier) paired with their respective scientific names (`scientificName`).168- `metadata_500.csv`: 500 image subset of `metadata_10k.csv` for trait detection and counting. `ARKID` is unique identifier from Fish-AIR, links to full metadata information in `full_fish_metadata.csv`.169- `processed_identification_imagelist_10k.txt`: List of images included in the `processed_identification_matrix.csv`.170- `processed_identification_matrix.csv`: Presence/Absence indicator for 10 external (visible) traits: `eye`, `head`, `mouth`, `barbel`, `dorsal fin`, `two dorsal fins`, `adipose fin`, `pectoral fin`, `pelvic fin`, `anal fin`. Unique identifier is the `fileNameAsDelivered`, and scientific name is indicated (`scientificName`).171 172**Bird Files:**173- `bird_imagelist_10k.txt`: List of image filenames for species classification and trait identification.174- `bird_metadata_10k.csv`: Image filenames (`fileNameAsDelivered`, unique identifier) paired with their respective scientific names (`scientificName`).175- `identification.csv`:176- `processed_identification.csv`:177- `trait_category_map.pkl`:178 179**Butterfly Files:**180- `imagelist.txt`: List of image filenames for species classification.181- `metadata.csv`: Image filenames (`fileNameAsDelivered`, unique identifier) paired with their respective scientific names (`scientificName`).182 183VLM Prompts are determined through code available in the [GitHub Repository](https://github.com/Imageomics/VLM4Bio); they are summarized in the task diagram above.184 185### Data Splits186 187These images were all used for benchmarking current state-of-the-art VLMs on biological tasks.188 189### Curation Rationale190<!-- Motivation for the creation of this dataset. For instance, what you intended to study and why that required curation of a new dataset (or if it's newly collected data and why the data was collected (intended use)), etc. -->191 192 193### Source Data194 195We collected images of three taxonomic groups of organisms: fish, birds, and butterflies, each containing around 10k images. 196 197#### Fish198 199Images for fish (Fish-10k) were curated from the larger image collection, [Fish-AIR](https://fishair.org/), which contains images from the [Great Lakes Invasives Network (GLIN)](https://greatlakesinvasives.org/portal/index.php).200 201These images originate from various museum collections, including the following:202 203- [Illinois Natural History Survey (INHS)](https://biocoll.inhs.illinois.edu/portal/index.php)204- [Minnesota Biodiversity Atlas, Bell Museum](https://bellatlas.umn.edu/index.php)205- [University of Michigan Museum of Zoology (UMMZ), Division of Fishes](https://ipt.lsa.umich.edu/resource?r=ummz\_fish)206- [University of Wisconsin-Madison Zoological Museum - Fish](http://zoology.wisc.edu/uwzm/)207- [Field Museum of Natural History (Zoology, FMNH) Fish Collection](https://fmipt.fieldmuseum.org/ipt/resource?r=fmnh_fishes)208- [The Ohio State University Fish Division, Museum of Biological Diversity (OSUM), Occurrence dataset](https://doi.org/10.15468/subsl8)209 210[Phenoscape](https://kb.phenoscape.org/about/phenoscape/kb) and [FishBase](https://www.fishbase.se/search.php) were used to obtain the information on traits. 211 212- **Data Processing:**213 214  We created the Fish-10k dataset by randomly sampling 10K images and preprocessing the images to crop and remove the background. For consistency, we leverage [GroundingDINO](https://arxiv.org/abs/2303.05499) to crop the fish body from the background and [Segment Anything Model (SAM)](https://arxiv.org/abs/2304.02643) to remove the background.215  This is the same processing done in [Fish-Vista](https://huggingface.co/datasets/imageomics/fish-vista), more details and the code is available [here](https://github.com/sajeedmehrab/Fish-Vista/tree/main/code/processing). 216 217 218#### Bird219 220We create the Bird-10k dataset from the [CUB-200-2011 dataset](https://www.vision.caltech.edu/datasets/cub_200_2011/). We obtain the scientific names from the [iNatLoc dataset](https://data.caltech.edu/records/ts54e-1jr46). 221 222- **Data Processing:**223 224  For Bird-10k, we take 190 species for which the common name to scientific name mapping is available. This results in a fairly balanced dataset. Please download the images following the directions under [Dataset Structure](#dataset-structure).225 226 227#### Butterflies228 229We created the Butterfly-10k dataset from the [Heliconius Collection (Cambridege Butterfly) dataset](https://huggingface.co/datasets/imageomics/Heliconius-Collection_Cambridge-Butterfly).230 231- **Data Processing:**232 233  For the Butterfly-10k, we carefully sampled 10K images from the Heliconius Collection dataset to ensure the images capture unique specimens and represent a diverse set of species. We adopt the following steps:234 235  - We filter out images with more than one image from the same view (i.e., dorsal or ventral).236  - We ensure each species has a minimum of 20 images and no more than 2,000 images.237 238 239### Annotations240 241##### Scientific Names242The scientific names for the images of **Fish-10k** and **Butterfly-10k** were obtained directly from their respective sources.243 244For **Bird-10k**, we obtained the scientific names from the [iNatLoc dataset](https://data.caltech.edu/records/ts54e-1jr46). 245 246In total, we curated around 31K question-answer pairs in both open and multiple-choice (MC) question-formats for evaluating species classification tasks. 247 248##### Trait information249The species-level trait presence/absence matrix for Fish-10k was manually curated with the help of biological experts co-authored in this paper. We leveraged the [Phenoscape knowledge base](https://kb.phenoscape.org/about/phenoscape/kb) and [FishBase](https://www.fishbase.se/search.php) along with manual annotations to procure the presence-absence trait information. We constructed approximately xK question-answer pairs for Fish-10k250 251For Bird-10k, we obtained the trait matrix from the attribute annotations provided along with CUB-200-2011. 252 253In total, we constructed approximately 380K question-answer pairs for trait identification tasks. 254 255##### Grounding and referring256For grounding and referring VQA tasks, the ground truths were manually annotated with the help of expert biologists on our team. We manually annotated bounding boxes corresponding to the traits of 500 fish specimens and 500 bird specimens, which are subsets of the larger Fish-10k and Bird-10k datasets, respectively. 257We used the CVAT tool for annotation.258 259### Personal and Sensitive Information260 261None262 263## Considerations for Using the Data264 265The fish-10 K and Butterfly-10K datasets are not balanced for the species classification task, while the bird-10 K dataset is balanced. Since the fish images are collected from different museums, they may inherit a small bias. 266 267## Licensing Information268 269This dataset (the compilation) has been licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0). However, images may be licensed under different terms (as noted above). For license and citation information by image, see our [license file](https://huggingface.co/datasets/sammarfy/VLM4Bio/blob/main/licenses.csv).270 271 - The fish images are from [GLIN](https://greatlakesinvasives.org/portal/index.php), whose metadata and source URLs were accessed through [Fish-AIR](fishair.org). The GLIN images are all either [CC BY-NC](https://creativecommons.org/licenses/by-nc/4.0/) or in the public domain ([CC0](https://creativecommons.org/publicdomain/zero/1.0/)).272 - All the bird images are sourced from the [CUB-200-2011 dataset](https://www.vision.caltech.edu/datasets/cub_200_2011/); CalTech states that they do not own the copyrights to these images, and that their use is restricted to non-commercial research and educational purposes.273 - All butterfly images are from the Butterfly Genetics Group at University of Cambridge and are licensed under [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/).274 275Each image in this dataset is provided under the least restrictive terms allowed by its licensing requirements as provided to us (i.e., we impose no additional restrictions past those specified by licenses in the license file).276 277We provide licensing information for every individual image within the fish and butterfly images in [license-metadata/fish-licenses.csv](license-metadata/fish-licenses.csv) and [license-metadata/butterfly-licenses.csv](license-metadata/butterfly-licenses.csv) respectively. The source_link and citation for each of the butterfly images can be obtained by matching the `record_number` field to the record numbers in [license-metadata/butterfly-licenses.json](license-metadata/butterfly-licenses.json). 278 279## Citation280Please cite our work as follows:281 282```283@misc{maruf2024vlm4bio,284      title={VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images}, 285      author={M. Maruf and Arka Daw and Kazi Sajeed Mehrab and Harish Babu Manogaran and Abhilash Neog and Medha Sawhney and Mridul Khurana and James P. Balhoff and Yasin Bakis and Bahadir Altintas and Matthew J. Thompson and Elizabeth G. Campolongo and Josef C. Uyeda and Hilmar Lapp and Henry L. Bart and Paula M. Mabee and Yu Su and Wei-Lun Chao and Charles Stewart and Tanya Berger-Wolf and Wasila Dahdul and Anuj Karpatne},286      year={2024},287      eprint={2408.16176},288      archivePrefix={arXiv},289      primaryClass={cs.CV},290      url={https://arxiv.org/abs/2408.16176}, 291}292```293 294Please be sure to **also cite the original data sources** using all of the citations provided in the following:295 296- Sources for Fish-10k: [license-metadata/fish-data-bib.bib](license-metadata/fish-data-bib.bib).297- Sources for Bird-10k: [license-metadata/bird-data-bib.bib](license-metadata/bird-data-bib.bib)298- Sources for Butterfly-10k: [license-metadata/butterfly-data-bib.bib](license-metadata/butterfly-data-bib.bib)299 300## Acknowledgements301 302This work was supported by the Imageomics Institute, which is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under Award #2118240 (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.303 304 305## Dataset Card Authors306 307M. Maruf, Kazi Sajeed Mehrab and Elizabeth G. Campolongo308 309## Dataset Card Contact310 311marufm@vt.edu, ksmehrab@vt.edu312 313 314 315