aymanbm2000/ag-news-distilbert-lora-r4

AG News Topic Classification — DistilBERT + LoRA (r=4, alpha=8)

A DistilBERT model fine-tuned with LoRA (via PEFT) for 4-class topic classification on the AG News dataset: World, Sports, Business, Sci/Tech.

This is one of four LoRA ranks (r=4/8/16/32) trained and compared against a TF-IDF+Logistic Regression baseline and a fully fine-tuned DistilBERT, as part of a broader project exploring the performance/cost tradeoff of LoRA rank on a short-text classification task. Full comparison, training pipeline, and code: github.com/aymanbm2000/project3

Model description

• —Base model: distilbert-base-uncased
• —Fine-tuning method: LoRA, rank=4, alpha=8, dropout=0.1
• —Target modules: q_lin, v_lin (attention query/value projections)
• —Trainable parameters: a small fraction of DistilBERT's 67M parameters (adapter + classifier head only) — the smallest adapter in this rank sweep
• —Training data: ~50,000 rows sampled from AG News' training split
• —Epochs: 3

Intended use

Topic classification of short English news-style text into one of four categories. This is the lowest-rank, most lightweight adapter in the sweep — a good fit where adapter size/training cost matters most and a small F1 tradeoff vs. higher ranks is acceptable.

Not intended for: text outside the news-topic domain, languages other than English, or fine-grained/subtopic classification beyond the four AG News categories.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel
import torch

base_model_name = "distilbert-base-uncased"
adapter_repo = "aymanbm2000/ag-news-distilbert-lora-r4"

tokenizer = AutoTokenizer.from_pretrained(adapter_repo)
base_model = AutoModelForSequenceClassification.from_pretrained(base_model_name, num_labels=4)
model = PeftModel.from_pretrained(base_model, adapter_repo)
model.eval()

text = "The central bank raised interest rates for the third consecutive quarter."
inputs = tokenizer(text, truncation=True, max_length=128, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]
    pred_idx = int(torch.argmax(probs))

label_names = ["World", "Sports", "Business", "Sci/Tech"]
print(label_names[pred_idx], float(probs[pred_idx]))

Evaluation results

Evaluated on the standard AG News test split (7,600 examples).

Comparison across ranks and against baseline/full fine-tuning

Even at the lowest rank tested, LoRA already recovers most of full fine-tuning's benefit over the TF-IDF baseline. Higher ranks improve F1 further but with diminishing, incremental gains.

Training procedure

• —Optimizer/scheduler: HuggingFace Trainer defaults
• —Learning rate: 2e-4 (LoRA — higher than full fine-tuning's 2e-5, per common LoRA practice)
• —Batch size: 32 (train) / 64 (eval)
• —Max sequence length: 128 tokens
• —Selection: best checkpoint by macro F1 on a held-out validation split (10% of training data)
• —Tracked with Weights & Biases

Limitations

• —Trained and evaluated only on AG News; performance on other news-classification datasets or domains is untested.
• —English only.
• —Like the base model, may reflect biases present in DistilBERT's pretraining data.
• —As the lowest-rank adapter in this sweep, has the smallest capacity of the four LoRA configs — if higher accuracy matters more than adapter size, consider r=16 or r=32 instead.