xn6o/lora-vit-large-patch16-224-in21k-r32-imagenet1k

🌟 ViT-L/16 or ViT-B/16 LoRA — ImageNet-1K Fine-tune (Top-1 ≈ 71%)

This repository contains LoRA fine-tuned weights for Vision Transformer (ViT) models trained on the ImageNet-1K classification dataset.

The repository includes:

• —LoRA adapter weights (query/key/value/output + MLP LoRA, r=32)
• —Classification head weights (classifier_head.pth)
• —A detailed model card (this file)

This enables loading the pretrained ViT together with LoRA and the final classifier head to reproduce the reported ImageNet-1K accuracy.

📌 Model Summary

🚀 How to Use

1️⃣ Load the base ViT model

from transformers import ViTForImageClassification
from peft import PeftModel
import torch

base = ViTForImageClassification.from_pretrained(
    "google/vit-base-patch16-224"
)

2️⃣ Load LoRA adapter

model = PeftModel.from_pretrained(
    base,
    "username/repo-name"
)

3️⃣ Load classification head

state = torch.load("classifier_head.pth", map_location="cpu")
model.base_model.model.classifier.load_state_dict(state)
model.eval()

🎯 Intended Use

This model is designed for:

• —ImageNet-1K classification
• —Downstream dataset finetuning via LoRA
• —Feature extraction / embedding extraction
• —Transfer learning to custom datasets
• —Model compression and deployment

It is not intended for:

• —Safety-critical systems
• —Medical or legal decision making
• —Bias-sensitive applications

🧩 Training Details

LoRA Configuration

{
  "r": 32,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "target_modules": [
    "query", "key", "value", "output.dense",
    "intermediate.dense", "output.dense"
  ]
}

LoRA is applied to:

• —Multi-head self-attention Q/K/V/O
• —MLP hidden layer (to/from dim 4096 → 1024)

Optimizer & Schedule

• —Optimizer: AdamW
• —Learning rate: 1e-4
• —Weight decay: 0.01
• —LR scheduler: cosine annealing
• —Warmup: None

📚 Dataset

• —ImageNet-1K (ILSVRC2012)
• —1.28M train images
• —50k validation images
• —1000 classes

🥇 Evaluation

Evaluation was done on the official 50k-val set.

📦 Files Included

adapter_model.safetensors   # LoRA weights
adapter_config.json         # LoRA configuration
classifier_head.pth         # Final classification head
README.md                   # This model card

📝 Citation

If you use this model, please cite:

@article{hu2021lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward J. and others},
  year={2021}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and others},
  year={2020}
}

⚠️ Limitations

• —Trained only for 224×224 resolution
• —May be biased to ImageNet categories
• —LoRA updates only ~2% of parameters (some categories could underfit)

⚖ License

This model inherits the license of the base ViT model and LoRA implementation. Check the respective repos for details.

🙌 Acknowledgements

Thanks to:

• —Google Research (ViT)
• —Hugging Face Transformers
• —PEFT Team (LoRA)
• —ImageNet dataset maintainers