tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-SpectralSurgery-HNS8p2-AllMods

Qwen2.5-7B-Instruct + Commonsense170K — Spectral Surgery HNS 8+2

This repository contains a post-hoc Spectral Surgery adapter derived from the Qwen2.5-7B-Instruct Commonsense170K LoRA checkpoint. No additional gradient-based training is performed during Spectral Surgery.

Source LoRA

• —Source: tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-LoRA
• —Dataset: Commonsense170K, 170,420 examples
• —Epochs: 2 (10,652 optimizer steps)
• —Sequence length: 2,048
• —Global batch size: 32
• —Learning rate: 2e-4
• —LoRA rank/alpha/dropout: 16 / 32 / 0.05
• —Target modules: all seven LoRA projection types
• —Seed: 42

Spectral Surgery

• —Method: Hybrid Newton-Schulz (HNS)
• —Scope: all 196 LoRA modules
• —Fast/stable steps: 8 + 2
• —Output rank: 16
• —Nuclear norm: preserved
• —Mean effective rank: 11.1021 → 16.0000

Exact per-module statistics and coefficients are recorded in spectral_edit_meta.json.

Evaluation

Greedy evaluation on the eight-task commonsense suite, using the tokenizer chat template in non-thinking mode and at most 8 generated tokens.

HNS changes macro accuracy by +0.1801 percentage points relative to the source LoRA. Its micro accuracy changes by -0.1829 points (-41 correct answers), so the effect is a redistribution across tasks rather than a uniform improvement.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(
    base,
    "tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-SpectralSurgery-HNS8p2-AllMods",
)

Files

• —adapter_model.safetensors / adapter_config.json: edited PEFT adapter
• —spectral_edit_meta.json: exact HNS configuration and per-module statistics
• —run_args.json, run_config.json, training_args.json: source training configuration
• —eval-commonsense8/: aggregate metrics and per-example HNS predictions
• —comparison-summary.json / .tsv: Base, LoRA, and HNS comparison