scatyf3/speccompact-draft-sft-ep2-qwen3-14b-distill

speccompact-draft-sft-ep2-qwen3-14b-distill

LoRA adapter for Qwen/Qwen3-0.6B, trained as a speculative-decoding draft model for context-compaction ("slot summary") generation in the SpecCompact project.

The adapter is fine-tuned via supervised distillation on slot-summary targets (y*) generated by a Qwen3-14B teacher, so the 0.6B draft learns to approximate the teacher's compaction outputs closely enough to be used as the draft model in speculative decoding against a larger target model.

This checkpoint is epoch 2 of that SFT run (resumed from an epoch-1 checkpoint of an earlier job with the same config).

Training details

• —Base model: Qwen/Qwen3-0.6B
• —Method: LoRA, rank=16, alpha=32, dropout=0.05, target modules q_proj,k_proj,v_proj,o_proj
• —Teacher / distillation target: Qwen3-14B slot-summary rollouts
• —Training data: 58,140 examples (slot_targets_qwen3_14b_sft_combined.jsonl)
• —Optimizer: lr=1e-4, batchsize=8, gradaccum_steps=64 (effective batch 512)
• —Context length: 8192 tokens, chat-template formatted
• —Final epoch-2 cross-entropy: 0.468

Usage

Load as a PEFT LoRA adapter on top of Qwen/Qwen3-0.6B, e.g. served via vLLM with --enable-lora --lora-modules draft=<local checkpoint path>, or with transformers/peft:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
model = PeftModel.from_pretrained(base, "scatyf3/speccompact-draft-sft-ep2-qwen3-14b-distill")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

Framework versions

• —PEFT 0.19.1