tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-LoRA

Qwen2.5-7B-Instruct + Commonsense170K — LoRA

This repository contains the LoRA adapter produced by supervised fine-tuning Qwen2.5-7B-Instruct on Commonsense170K. It is the source adapter for the corresponding post-hoc Spectral Surgery HNS 8+2 experiment.

Training

• —Base model: Qwen/Qwen2.5-7B-Instruct
• —Dataset: Commonsense170K, 170,420 examples
• —Epochs: 2 (10,652 optimizer steps)
• —Sequence length: 2,048
• —Global batch size: 32
• —Learning rate: 2e-4, cosine schedule, warmup ratio 0.10
• —LoRA rank/alpha/dropout: 16 / 32 / 0.05
• —Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• —Precision: bf16
• —Seed: 42
• —Chat template: non-thinking

Evaluation

Greedy evaluation on the eight-task commonsense suite, using the tokenizer chat template in non-thinking mode and at most 8 generated tokens.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(
    base,
    "tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-LoRA",
)

Files

• —adapter_model.safetensors / adapter_config.json: final PEFT adapter
• —run_args.json, run_config.json, training_args.json: training configuration
• —eval-commonsense8/: aggregate metrics and per-example LoRA predictions
• —comparison-summary.json / .tsv: Base, LoRA, and HNS comparison