dougalldeepmind/2026-07-29-qwen36-difficult-advice-tulu-lora-40-60
Qwen3.6-27B — difficult-advice + TULU3 LoRA (40/60 mixture)
LoRA adapter for `Qwen/Qwen3.6-27B`, trained on 40% difficult-advice / 60% TULU3 by token count.
One arm of a mixture-ratio sweep. The 20/80 sibling (`LASR-Callum/2026-07-28-qwen36-difficult-advice-tulu-lora-20-80`) cut ODCV-Bench misalignment from 37.2% → 19.2% against a matched FP8 base. The sweep asks whether more difficult-advice data strengthens that effect, plateaus, or tips into over-refusal.
All three hold total tokens at ~1.493M, so step count and compute are matched and the mixture ratio is the only variable.
Training mixture
Qwen3.6's chat template renders <think>{reasoning}</think> for any assistant turn that is final, so trace-free replay data would emit an empty <think></think> — the documented failure mode that trains a model to stop reasoning. The mixture builder appends a throwaway user turn to push the assistant off the end, hitting the template's no-think branch, then strips it. Verified on the written artifact: zero empty think blocks, think blocks in exactly the 580 difficult-advice rows.
Training
bf16 LoRA (not QLoRA — bitsandbytes does not reliably cover this model's hybrid linear-attention/SSM layers), 1×H100 80GB, 1h37m.
Packing is off because TRL only guarantees packed-sequence isolation under Flash Attention variants; under sdpa it warns of cross-contamination between samples. The vision tower (model.visual) is untouched.
Loss: 2.69 → ~1.00 by step 20, then flat (0.91–1.08). Final token accuracy 0.708, grad_norm 0.31, 1,444,984 tokens consumed.
Status
Not yet evaluated on ODCV-Bench. When it is, the comparison will be against the same matched FP8 base arm (37.2%) used for the 20/80 result, on the same 78 scenario cells with the same two judges.
Usage
from peft import PeftModel
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-07-29-qwen36-difficult-advice-tulu-lora-40-60")
model = model.merge_and_unload() # vLLM LoRA support for this hybrid arch is unprovenUse AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language checkpoint. Merging drops the base model's 15 mtp.* tensors, so speculative decoding needs them grafted back from the base checkpoint.