bioscan-ml/clibd

1---2description: "Pretrained weights for CLIBD, a multimodal model bridging vision and genomics for biodiversity monitoring."3---4 5# Model Card for CLIBD6 7In this model repo we provide the official pretrained models used in the paper **CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale.**8The model usage and code can be found in the [github repo](https://github.com/bioscan-ml/clibd).9 10## Model Details11 12### Model Description13 14- **Finetuned from model:**15 16    -Image: timm model (["vit_base_patch16_224"](https://huggingface.co/timm/vit_base_patch16_224.mae))17 18    -DNA barcode: BarcodeBERT ["bioscanr/barcodeBERT pre-trained on CANADA-1.5M"](https://huggingface.co/bioscan-ml/bioscan-clibd/tree/main/ckpt/BarcodeBERT/5_mer)19 20    -Text: Pre-trained BERT model  (["prajjwal1/bert-small"](https://huggingface.co/prajjwal1/bert-small))21### Model Sources22 23- **Repository:** https://github.com/bioscan-ml/clibd24- **Paper:** https://arxiv.org/abs/2405.1753725 26### Model Checkpoints27 28- **ckpt/bioscan_clip/final_experiments/image_dna_4gpu_50epoch/best.pth:** The model trained on the BIOSCAN-1M dataset by aligning images and DNA.29- **ckpt/bioscan_clip/final_experiments/image_dna_text_4gpu_50epoch/best.pth:** The model trained on the BIOSCAN-1M dataset by aligning images, DNA, and taxonomy labels.30- **ckpt/bioscan_clip/new_5M_training/image_dna_4gpu_50epoch/best.pth:** The model trained on the BIOSCAN-5M dataset by aligning images and DNA.31- **ckpt/bioscan_clip/new_5M_training/image_dna_text_4gpu_50epoch/best.pth:** The model trained on the BIOSCAN-5M dataset by aligning images, DNA, and taxonomy labels.32 33## Training Data34 35-[BIOSCAN-1M](https://huggingface.co/datasets/bioscan-ml/BIOSCAN-1M).36 37-[BIOSCAN-5M](https://huggingface.co/datasets/bioscan-ml/BIOSCAN-5M).38 39You can also find the processed data from [here](https://huggingface.co/datasets/bioscan-ml/bioscan-clibd).40 41**BibTeX:**42```bibtex43@article{gong2024clibd,44  title={{CLIBD}: Bridging Vision and Genomics for Biodiversity Monitoring at Scale},45  author={Gong, ZeMing and Wang, Austin T. and Huo, Xiaoliang and Haurum, Joakim Bruslund and Lowe, Scott C. and Taylor, Graham W. and Chang, Angel X.},46  journal={arXiv preprint arXiv:2405.17537},47  year={2024},48  eprint={2405.17537},49  archivePrefix={arXiv},50  primaryClass={cs.AI},51  doi={10.48550/arxiv.2405.17537},52}53```54 55## Acknowledgement56 57We would like to express our gratitude for the use of the INSECT dataset, which played a pivotal role in the completion of our experiments. Additionally, we acknowledge the use and modification of code from the [Fine-Grained-ZSL-with-DNA](https://github.com/sbadirli/Fine-Grained-ZSL-with-DNA) repository, which facilitated part of our experimental work. The contributions of these resources have been invaluable to our project, and we appreciate the efforts of all developers and researchers involved.58 59This reseach was supported by the Government of Canada’s New Frontiers in Research Fund (NFRF) [NFRFT-2020-00073],60Canada CIFAR AI Chair grants, and the Pioneer Centre for AI (DNRF grant number P1).61This research was also enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca).