Modelpublic

moolm/candle-qwen38-27b-sft-1p

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

1likes27downloads

Model Card

moolm Qwen3.8-27B SFT (LoRA)

QLoRA adapter that teaches Qwen/Qwen3.8-27B to write, explain, and refactor MOOCode, reason about Torchship design, and drive the live #340 agent harness (tool calls).

This is an adapter-only release (~897 MB). Load it on top of the base model; it is not a full merge.

Intended use

—MOOCode authoring, explanation, review, refactor, and dialect translation (mooR / classic LambdaMOO-style)
—Agentic multi-turn tool use against Torchship-style MCP tool schemas
—Chat-style design / principles / conventions Q&A grounded in the moolm corpus

Out of scope: general-purpose chat replacement for the base model; vision/video (this SFT is text/tool oriented); unsupervised production writes without a sandbox.

Training data

Built by the moolm pipeline from mined Torchship sessions plus synthesized, compile-checked examples.


Source file	`sft.reasoned.jsonl`
Rows in file	18,485
Kept for this run	7,946 complete examples with sequence length in `[1, 4096]`
Dropped	3,135 over-budget rows (never truncated)
Tracks (full file)	~57% agentic / ~43% chat
Dialects (full file)	~99% mooR, ~1% LambdaMOO
Sources (full file)	Torchship, Sindome, HellMOO, FallMOO

Training only supervised the weight=1 assistant turn in each window. Tool schemas were omitted from the chat template (tool calls retained). Nearly all rows carry reasoning_content from the reasoning backfill.

Training details

Hyperparameter	Value
Base	`Qwen/Qwen3.8-27B` (local text path; VL tower unused)
Method	QLoRA (NF4, double quant) + LoRA
Hardware	1× NVIDIA A100-SXM4-40GB
Max sequence	4096 (complete examples only)
LoRA rank / alpha / dropout	32 / 64 / 0.05
Target modules	`q/k/v/o_proj`, `gate/up/down_proj`, `in_proj_{a,b,qkv,z}`, `out_proj`
Trainable params	233,455,616 (~0.86%)
Epochs	2
Effective batch	16 (`per_device=1`, `grad_accum=16`)
LR	1e-4 cosine, warmup 70 steps
Optim	paged AdamW 8-bit
Precision	bf16 compute
Attention	SDPA
Seed	7
Final train loss	~0.575

Run artifact: training/output/qwen38-sft-full/final (completed 2026-08-16).

How to use

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_id = "Qwen/Qwen3.8-27B"
adapter_id = "moolm/candle-qwen38-27b-sft-1p"  # replace ORG

tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_id)

For 4-bit loading, pass a BitsAndBytesConfig when constructing the base model, then attach the adapter the same way.

Limitations

—Sequence budget for this adapter is 4096; longer agent windows from the dataset were excluded.
—Behavior is skewed toward Torchship / mooR; classic LambdaMOO coverage is thin.
—Agentic outputs assume Torchship #340-style tools; other harnesses need their own schemas in context.
—See Evaluation for held-out compile / tool metrics on A100-unused rows.

Evaluation

Held-out set drawn from the A100-unused slice of sft.reasoned.jsonl: complete examples whose training encoding exceeded 4096 tokens (never truncated into the SFT run). Stratified by family from the length band that still fits eval context.


Split	`a100_unused` (`qwen38_unused_eval.jsonl`)
Samples scored	128
Encoded length (train measure)	4203–12269 (median 9865)
Completions with ```moocode fences	49
Compile rate (moocheck)	98.0% (48/49)
Gold tool-call rows	41
Tool-name exact set match	0.0%
Mean tool-name F1	0.000
Mean content token Jaccard	0.202
Adapter	`/tank/data/Dev/moolm/training/output/qwen38-sft-full/final`
Eval timestamp (UTC)	2026-08-17T04:34:47.043036+00:00

Compile rate by family (rows with code fences)

family	ok/n	rate
agentic_task	33/34	97.1%
agentic_turn	7/7	100.0%
principled_pushback	8/8	100.0%

Generation: greedy-ish sampling (temperature=0.3), thinking enabled then stripped for scoring; prompts capped so prompt+new tokens fit the eval context budget.

License

Apache 2.0, following the base Qwen3.8-27B license. You must also comply with Qwen's terms when redistributing derivatives.