tg-techie-agents/Q1D-4B-Blk0-V1
Q1D 4B · Block 0 · V1
A research artifact from Q1 Descent — reconstructing 1-bit "intelligence-density" recovery for open-weight models.
What it is. Qwen3-4B with transformer block 0 quantized to 1-bit (Q1_0_g128: 1.125 bpw, a sign per weight + one FP16 scale per 128-group) and then trained back toward the full-precision teacher. Blocks 1–35 remain F16. So it isolates a single 1-bit block in an otherwise-F16 model — a validation of the recovery recipe, not a fully 1-bit model.
Recipe (V1). Straight-through estimator: FP32 master weights, forward = the exact Q1_0 encoding (so both signs and scales move), backward = straight through. Objective: KL to the FP16 teacher on C4 — 400 steps, lr 1e-4, bf16. The trained weights are the deployed 1-bit encoding (no latent/deploy gap).
Result (GSM8K, greedy, thinking-off):
Training recovers the dented block to ≈ full-precision capability at 1 bit.
Use. llama.cpp / LM Studio. Ships a closed-think (no-think) chat template; greedy (temperature 0) recommended.
hf download tg-techie-agents/Q1D-4B-Blk0-V1Caveats. Early research artifact: one 1-bit block (not a full 1-bit model), 4B scratchpad scale, single seed, GSM8K-only. Known token-level instability in some contexts (an occasional malformed token) — under investigation. Not affiliated with PrismML or the Qwen team.