Modelpublic

merci1ess/birdclef2026-solution

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

Model Card

🐦 BirdCLEF 2026 — Improved Solution for Production Robustness

Problem: Model performs well on training data but fails in production (passive acoustic monitoring).

Root cause: Focal → soundscape domain shift (6 identified causes).

Solution: Bird-MAE-Large backbone + EfficientNet-B1 ensemble with aggressive domain adaptation.

Architecture

Model	Backbone	Params	Role	Ensemble Weight
Bird-MAE-Large	ViT-L/16 (SSL on 1.7M Xeno-Canto)	302M	Primary	60%
EfficientNet-B1	EfficientNet (ImageNet)	19M	Diversity	40%

Key Anti-Overfitting Techniques

Addressing Domain Shift (focal → soundscape)

Technique	Probability	Source Paper	Impact
Waveform mixup (up to 3 sources)	0.9	Bird-MAE (2504.12880)	+29pp mAP
Background noise injection	0.5	Bird-MAE	Simulates PAM conditions
Colored noise	0.2	Bird-MAE	Different mic responses
Cyclic rolling	1.0	Bird-MAE	Removes position bias
Gain adjustment	0.2	Bird-MAE	Recording distance variation
SpecAugment (freq=50, time=100)	0.3	Bird-MAE	Regularization

Addressing Noisy/Weak Labels

Technique	Source	Why
Asymmetric Loss (γneg=4, γpos=0, clip=0.05)	ASL (2009.14119)	Down-weights easy negatives from missing background labels
Energy-peak window selection	Perch 2.0 (2508.04665)	Selects windows with actual bird vocalizations

Addressing Class Imbalance

Technique	Why
Weighted random sampling	Equal species exposure regardless of sample count
cmAP-aware design	Every class matters equally in the evaluation metric

Training Stability

Technique	Value	Source
Layer-wise LR decay	0.75	Bird-MAE Table 10
2-stage training	2 epochs frozen → 28 full	sl-BEATs (2508.11845)
Gradient clipping	2.0	Bird-MAE
Warmup	3 epochs	Standard
Cosine annealing	Full schedule	Standard

Post-Processing (Inference)

—No-call detection (suppress low-confidence segments)
—Probability thresholding (0.01)
—Multi-fold ensemble (5-fold CV × 2 architectures = 10 models)
—Test-time augmentation (time-reversal + gain)

Quick Start

bash

# 1. Download BirdCLEF 2026 data from Kaggle
kaggle competitions download -c birdclef-2026 -p ./data
unzip ./data/birdclef-2026.zip -d ./data/birdclef2026

# 2. Train Bird-MAE-Large (all 5 folds)
for fold in 0 1 2 3 4; do
  python train_birdmae.py \
    --data_dir ./data/birdclef2026/train_audio \
    --metadata ./data/birdclef2026/train_metadata.csv \
    --taxonomy ./data/birdclef2026/taxonomy.csv \
    --output_dir ./outputs/birdmae \
    --hub_model_id YOUR_USER/birdclef2026-birdmae \
    --fold $fold --epochs 30 --batch_size 32
done

# 3. Train EfficientNet-B1 (all 5 folds)
for fold in 0 1 2 3 4; do
  python train_effnet.py \
    --data_dir ./data/birdclef2026/train_audio \
    --metadata ./data/birdclef2026/train_metadata.csv \
    --taxonomy ./data/birdclef2026/taxonomy.csv \
    --output_dir ./outputs/effnet \
    --hub_model_id YOUR_USER/birdclef2026-effnet \
    --fold $fold --epochs 50 --batch_size 64
done

# 4. Inference + Ensemble
python inference.py --test_dir ./data/test_soundscapes --model_dir ./outputs/birdmae --output sub_mae.csv --tta
python inference.py --test_dir ./data/test_soundscapes --model_dir ./outputs/effnet --output sub_effnet.csv --tta
python ensemble.py --submissions sub_mae.csv sub_effnet.csv --weights 0.6 0.4 --output final.csv

Hardware Requirements

Model	GPU	VRAM	Time/fold
Bird-MAE-Large	A100 80GB	~40GB	~6-8h
EfficientNet-B1	A10G 24GB	~8GB	~3-4h

Dependencies

torch>=2.0
torchaudio>=2.0
transformers>=4.45,<5
librosa
scikit-learn
pandas
numpy
soundfile
trackio
huggingface_hub
hf_transfer

References

1.Bird-MAE: 2504.12880 — SSL pretraining, augmentations (+29pp mAP)
2.sl-BEATs-all: 2508.11845 — Best OOD generalization (0.01 AUROC drop focal→soundscape)
3.Perch 2.0: 2508.04665 — Energy-peak selection, self-distillation
4.Asymmetric Loss: 2009.14119 — Handles noisy multi-label
5.BirdSet: 2403.10380 — Multi-task benchmark
6.FINCH: 2602.03817 — Spatiotemporal fusion