tianzl66/Qwen3-8B-InstructionFollowing-SpectralSurgery-HNS4p1

Qwen3-8B + Instruction Following + Spectral Surgery

This repository contains a Spectral Surgery adapter derived from the all-linear instruction-following LoRA checkpoint of Qwen3-8B.

Post-hoc Spectral Surgery applies Hybrid Newton-Schulz (HNS) editing directly to the LoRA adapter. No additional gradient-based training is performed during Spectral Surgery.

Base Model

Qwen/Qwen3-8B

Source LoRA

• —Dataset: Tulu-3 SFT instruction-following split
• —Samples: 29,980
• —Epochs: 2
• —Sequence length: 4096
• —Global batch size: 128
• —LoRA rank: 16
• —LoRA alpha: 32
• —LoRA dropout: 0.05
• —Learning rate: 4e-4
• —LR schedule: cosine, warmup ratio 0.03
• —Target modules: qproj, kproj, vproj, oproj, gateproj, upproj, down_proj
• —Chat template: non-thinking (enable_thinking=False)
• —Seed: 42

Spectral Surgery

• —Target: all LoRA modules
• —Modules edited: 252
• —Fast HNS steps: 4
• —Stable HNS steps: 1
• —Output rank: 16
• —Nuclear norm: preserved
• —Mean effective rank: 11.7908 → 15.9982 (4+1) / 16.0000 (8+2)
• —See spectral_edit_meta.json for exact edit metadata.

Evaluation

Evaluation on IFEval (541 prompts, 834 instructions).

Relative to the vanilla LoRA checkpoint, Spectral Surgery HNS 4+1 improves:

• —Prompt Strict by 1.48 percentage points
• —Prompt Loose by 2.40 percentage points
• —Instruction Strict by 1.80 percentage points
• —Instruction Loose by 1.56 percentage points

Compared with the base model, this HNS edit improves both strict metrics; the two loose metrics remain slightly below base.

Settings: Qwen3 non-thinking chat template (enable_thinking=False), greedy decoding, max_new_tokens=2048, vLLM backend, FLASH_ATTENTION, max model length 4096, seed 42.

Files

• —adapter_model.safetensors: PEFT LoRA weights
• —adapter_config.json: PEFT configuration
• —eval-ifeval/metrics.json: aggregate IFEval metrics
• —eval-ifeval/outputs.jsonl: per-prompt generations and instruction results
• —spectral_edit_meta.json: exact HNS edit metadata