milwright/cloze-reader-qwen3.5-0.8b-lora

cloze-reader-qwen3.5-0.8b-lora

A rank-16 LoRA adapter over [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) that replaces a ~33× larger teacher (Gemma-3-27B-IT) as the language-model backend for the [Cloze Reader](https://reader.inference-arcade.com) reading-comprehension app (source).

On a 200-example held-out test set, the fine-tuned 0.8B student matches or exceeds the 27B teacher on 4 of 5 metric categories, with perfect format-constraint compliance on two tasks. The full model at bf16 is ~1.7 GB; at Q4KM it runs in ~2.5–3 GB of VRAM.

TL;DR

Update (2026-07): the live adapter has since had 6 additional continued-PEFT cycles of Gemini-distilled data on top of the original Gemma-only run. Training loss fell from 0.625 (original, 3 epochs) to 0.4304 (after cycle 6, ~20.3k examples). The tables below report the original n=200 Gemma-vs-student eval; the honest hint-safety figure under a 9-layer leak detector is ~72.5% no-leak (vs. the 98% a cheap exact-match check reports) — see Evaluation → Hint-leakage note. This adapter is the intended per-app LoRA for the shared inference-arcade vLLM host.

What it does — the four tasks

The Cloze Reader app uses a single LM for four tightly-constrained text-generation tasks. The adapter was trained on examples of all four and produces the exact JSON / plaintext shapes the front-end expects.

1. 1.Word Selection. Given a passage, pick 1–3 vocabulary words to blank. Output is a JSON array such as ["laboratory", "synthesis"]. Constraints: lowercase only, 4–14 letters, must appear verbatim in the passage, no proper nouns.
2. 2.Batch Word Selection. Same task across two passages at once. Output is a JSON object with per-passage word lists plus supporting context.
3. 3.Contextual Hints. Given a blanked word and its sentence, return a 15–25-word Socratic hint that points at part of speech, sentence role, or semantic category without revealing the target word.
4. 4.Literary Contextualization. One-sentence insight about a passage (≤25 words, no em-dashes, no verbose preamble like "This passage is about…").

Exact prompt templates match aiService.js / conversationManager.js in the cloze-reader repo.

How to use

With peft + transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

BASE = "Qwen/Qwen3.5-0.8B"
ADAPTER = "milwright/cloze-reader-qwen3.5-0.8b-lora"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, ADAPTER)

messages = [
    {"role": "system", "content": "Select words for a cloze exercise. Return ONLY a JSON array of words, nothing else."},
    {"role": "user", "content": "Select 1 challenging words (4-14 letters) from this passage.\n\nPassage: \"...\""},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tokenizer(prompt, return_tensors="pt").to(model.device), max_new_tokens=64)
print(tokenizer.decode(out[0], skip_special_tokens=True))

With Unsloth (4-bit, fast)

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    "milwright/cloze-reader-qwen3.5-0.8b-lora",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

With Ollama (via GGUF)

A merged GGUF build of this adapter fits ~2.5–3 GB at Q4\K\M and runs with:

ollama create cloze-reader -f Modelfile
ollama run cloze-reader

(The GGUF + Modelfile are not bundled here — see the project repo for the export pipeline.)

With vLLM (OpenAI-compatible server)

Standalone — serve the merged build (milwright/cloze-reader-qwen3.5-0.8b) directly:

vllm serve milwright/cloze-reader-qwen3.5-0.8b \
    --served-model-name cloze-reader \
    --host 127.0.0.1 --port 1234 \
    --dtype bfloat16 --max-model-len 2048

Shared base + adapter (multi-LoRA) — load this adapter on top of the base so one 0.8B model in VRAM can serve several app-specific adapters, each selected per request via the OpenAI model field:

vllm serve Qwen/Qwen3.5-0.8B \
    --enable-lora --max-lora-rank 16 \
    --lora-modules cloze-reader=milwright/cloze-reader-qwen3.5-0.8b-lora \
    --host 127.0.0.1 --port 1234 --max-model-len 2048

Then POST OpenAI-shape chat completions with "model": "cloze-reader". This is the serving mode used by the inference-arcade host, which attaches multiple sibling adapters to the same base.

Training

Data

• —Source: 15,981 filtered conversation examples (from 19,341 raw → 82.6% pass rate), distilled from `google/gemma-3-27b-it` serving the production cloze-reader endpoint.
• —Passages: Randomly-sampled text windows from 40 classic public-domain books via the `manu/project_gutenberg` corpus (61k books on the Hub).
• —Per-task composition (post-filter):
• —Word Selection — 7,433 examples
• —Batch Selection — 6,822 examples
• —Contextual Hints — 902 examples
• —Literary Contextualization — 824 examples
• —Format: ShareGPT-style {"conversations": [{"role": ..., "content": ...}, …]}, rendered through Qwen-3 ChatML (<|im_start|> / <|im_end|>).
• —Filter gates: JSON parsability, lowercase-only word selection, word-in-passage check, hint-safety (no leakage of the answer word), length bounds, em-dash / preamble interdiction.

Procedure

Supervised fine-tuning with TRL + Unsloth's FastLanguageModel, loss masked to assistant turns via train_on_responses_only().

Hyperparameters

Compute

• —Hardware: Single NVIDIA RTX 5090 Laptop GPU (24 GB VRAM).
• —Framework: Unsloth 2026.3.8 · TRL 0.24 · Transformers 5.3 · PEFT 0.18.1 · PyTorch 2.10 / CUDA 12.8.
• —Checkpoints saved: every epoch (checkpoint-1311, -2622, -3933).

Continued distillation (6 Gemini cycles)

After the original Gemma-only run, the adapter was refined with 6 additive continued-PEFT cycles (--resume-from, lr=5e-5, 1 epoch each). Each cycle adds +500/task of fresh data distilled from google/gemini-3.1-flash-lite-preview and merges it with the legacy corpus:

Loss kept falling ~linearly (no asymptote), but a 4-cell prompt × adapter matrix eval showed the structural metrics were already saturated and hint-leak safety did not move — the ceiling is structural to giving meaningful hints about content words, not a data-volume problem.

Evaluation

200 held-out examples (50 per task, seed 42, removed from training data pre-split) scored against the live Gemma-3-27B endpoint as baseline.

Word Selection (n=50)

Batch Selection (n=50)

Contextual Hints (n=50)

Literary Contextualization (n=50)

Hint-leakage note (cheap vs. layered detector)

The 98% "word safe" figure above comes from a cheap exact/substring match. A 9-layer detector (exact → normalized → substring → phonetic → lemma → synonym → hypernym → gloss-overlap → definition) that catches inflection, synonym, hypernym, and dictionary-gloss leaks measures ~72.5% no-leak on the n=40 holdout. The 22.5-point gap is dominated by gloss_overlap + hypernym events (≈75% of all leaks) and is structural: a hint that meaningfully describes a content-rich word tends to borrow words from its definition. Keep a heuristic safety filter downstream regardless of which number you quote.

Summary

• —Format / JSON compliance: the 0.8B student beats the 27B teacher on every structural metric. This is the usual distillation win — shape constraints fit inside a small parameter budget.
• —Content quality: near-parity. Word-selection validity lags by 2 points; hint word-safety lags by 2 points. Neither gap blocks production use in the cloze app.
• —Throughput / cost: ~33× fewer parameters, runs locally on a laptop GPU, removes the 27B API dependency.

Full per-metric JSON is at evaluation_results.json in the training project repo.

Intended use

In-scope. Serving the four cloze-reader tasks in the Cloze Reader app or a comparable vocabulary-practice / guided-reading UI, where inputs are short English prose passages (classical or modern) and outputs are JSON arrays / objects or tightly-length-bounded sentences.

Out of scope.

• —Open-ended generation, chat, or reasoning — this adapter has only seen 4 narrow instruction templates and will generalize poorly outside them.
• —Languages other than English — training data is English-only.
• —Safety-critical or factual-lookup tasks — no alignment or factuality work was performed beyond format-filtering.
• —Multimodal inputs — although Qwen3.5-0.8B is a vision-language model, this adapter was trained on text conversations only.

Limitations and risks

• —Distilled from a single teacher. Failure modes of gemma-3-27b-it on the 4 task prompts are inherited. If the teacher has a blind spot on certain passages (e.g., archaic or dialect text from Gutenberg), the student has the same one.
• —Gutenberg domain skew. Passages are drawn from ~40 classic public-domain books. Modern prose, social media, and non-narrative text are under-represented.
• —Format compliance is not correctness. "100% JSON valid" means the output parses; it does not guarantee the selected words are the pedagogically best choice. Human review is advised for educational deployment.
• —Hint-leakage floor of 2%. 1 in 50 hints referenced or strongly implied the target word in testing. Downstream code should keep a heuristic safety filter in place.

License

This adapter is released under Apache 2.0, inheriting from the base model Qwen/Qwen3.5-0.8B. Training data includes public-domain passages from Project Gutenberg and AI-generated outputs from google/gemma-3-27b-it; redistribution of the adapter weights themselves carries no Gutenberg restriction, but downstream users should honor Gemma's terms if they redistribute teacher generations separately.

Project context

This is the production LM for Cloze Reader, a reading-comprehension web app for practicing vocabulary through contextual word-blanking. Originally the app called a hosted Gemma-3-27B endpoint; this adapter was trained to bring inference on-device and retire the API dependency.

Developed as part of milwright/quimbot, a broader fine-tuning and evaluation project for small English-language models. See the repo's CLAUDE.md and fine-tuning/ for the larger pipeline.

Citations

Base model — Qwen Team, Qwen3.5-0.8B (2026), huggingface.co/Qwen/Qwen3.5-0.8B.

Teacher (training data) — Google DeepMind, Gemma 3 27B Instruct (2025), huggingface.co/google/gemma-3-27b-it.

Passage corpus — `manu/project_gutenberg`.

TRL

@misc{vonwerra2022trl,
  title  = {{TRL: Transformer Reinforcement Learning}},
  author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Galloué́́dec},
  year   = 2020,
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/huggingface/trl}}
}

Unsloth — github.com/unslothai/unsloth.

PEFT — Mangrulkar et al., PEFT: State-of-the-art Parameter-Efficient Fine-Tuning, github.com/huggingface/peft.

Framework versions

• —PEFT 0.18.1
• —TRL 0.24.0
• —Transformers 5.3.0
• —PyTorch 2.10.0 + CUDA 12.8
• —Unsloth 2026.3.8