TsinghuaCorals/bioclip-2.5-vith14-reefnet-lora

BioCLIP 2.5 ViT-H — ReefNet LoRA (species, 92-class global)

A LoRA fine-tune of `imageomics/bioclip-2.5-vith14` trained contrastively on the ReefNet 1.0 coral-reef species dataset (ReefNet/ReefNet-1.0), 92-class global curated split (train 48,312 / image-val 32,792 / image-test 33,090 / source-val 8,074; split cache 56ea94e36f9f).

Artifacts

This repo provides two equivalent forms:

• —`open_clip_model.safetensors` — the merged full model. Base BioCLIP 2.5 with the LoRA adapters folded into the weights and restored to the original nn.MultiheadAttention layout. Load as a plain CLIP, no PEFT needed:

  import open_clip
  model, _, preprocess = open_clip.create_model_and_transforms(
      "hf-hub:BobDerBaum/bioclip-2.5-vith14-reefnet-lora"
  )
  tokenizer = open_clip.get_tokenizer("hf-hub:BobDerBaum/bioclip-2.5-vith14-reefnet-lora")

• —`adapter.safetensors` + `adapter_config.json` — the raw PEFT LoRA adapters (vision encoder only, 3.6% trainable) plus the trained logit_scale. Load onto the base model with PEFT (see adapter_config.json for target_modules).

Training

• —Backbone: imageomics/bioclip-2.5-vith14 (frozen), LoRA on Q/K/V/O of attention and c_fc/c_proj of MLP of the vision encoder only (the text encoder was not adapted — text-encoder PEFT was not yet implemented at this run).
• —Rank 32, alpha 64, dropout 0.1; AdamW lr 1e-4, weight decay 0.01, focal loss (gamma 2.0) with class-frequency weights; 5% linear warmup + cosine annealing.
• —10 epochs (early stopped at epoch 5, best val_acc=0.8865).

Metrics (run 62yuzh9j)

(See the W&B run and the project's outputs/2026-06-27/01-27-46/ for full per-class metrics.)

Splits / Data

The training data and splits are published as `BobDerBaum/reefnet_species_images`.