xn6o/lora-vit-large-patch16-224-in21k-r96-imagenet1k

🧩 ViT-Large Patch16 224 - LoRA r=96 (ImageNet-1K Finetuned)

This repository contains a LoRA finetuned version of `google/vit-large-patch16-224-in21k` for ImageNet-1K classification, using rank = 96 LoRA adapters with additional data augmentation and strong regularization. The classifier head was retrained from scratch and is provided separately.

📌 Finetuning Summary

📂 Files in Repository

🚨 Important Notice

This LoRA model does not include classifier weights in the adapter. To reproduce correct evaluation accuracy, you MUST load both:

1️⃣ Base model 2️⃣ LoRA adapter 3️⃣ Custom classifier parameters

Failure to load the classifier will significantly reduce accuracy.

📦 Usage Example (PyTorch)

import torch
from transformers import ViTForImageClassification
from peft import PeftModel

device = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Load base model
model = ViTForImageClassification.from_pretrained(
    "google/vit-large-patch16-224-in21k",
    num_labels=1000
)

# 2. Load LoRA adapter
model = PeftModel.from_pretrained(model, "xn6o/lora-vit-large-patch16-224-in21k-r96-imagenet1k")

# 3. Load trained classifier weights
state_dict = torch.load("vit_classifier_r96.pt", map_location=device)
model.classifier.load_state_dict(state_dict)

model.to(device)
model.eval()

print("Model fully loaded with LoRA + classifier.")

🔍 Inference Example (Batch Images)

with torch.no_grad():
    outputs = model(pixel_values=images)
    preds = torch.argmax(outputs.logits, dim=-1)

📈 Training Details

Key techniques:

• —RandAugment + ColorJitter + RandomErasing
• —Cosine LR warm restart (iteration level)
• —AMP (BF16 preferred if supported)
• —Gradient clipping

📜 License

This model follows the same license as the original ViT base model from Google Research and Hugging Face Transformers. Usage must comply with ImageNet dataset license.

---


- PEFT 0.17.1