leapeto/mindcube-latent-ckpts

MindCube latent-reasoning checkpoints

LoRA adapters for Qwen/Qwen2.5-VL / Qwen2.5-VL-3B-Instruct that reason over MindCube spatial questions in a compressed latent vocabulary — the model emits <latent_start> <lat_…> <latent_end> <answer>X</answer> — instead of a natural-language chain of thought. Each <lat_i> is one BPE-merged unit standing for several base tokens, so the reasoning decodes in far fewer steps at no accuracy cost.

Training traces (text): `leapeto/mindcube-latent-data`. Images: `Inevitablevalor/MindCube` data.zip.

Checkpoints

Frozen-3B reference ≈ 37.8%; uncompressed full-CoT SFT baseline (plain_3b_ep5) = 46.6%. Accuracy is flat at ~49–51% across the whole 1.5×–4.05× compression sweep — compression is free down to the merge-table ceiling. Tinybench = MindCube's 1,050-question eval split, greedy two-stage decode.

The mn2 checkpoints are the high-compression extension: same recipe, trained on a regenerated (self-distilled) copy of the trace corpus. c2.0mn2 reproduces the original c2.0mn within noise (49.1% vs 51.1%) and exists only to confirm the corpora are comparable. Targets 4.5× and 5.0× both collapse to the same 4.05× vocabulary — the merge table runs out of recurring structure — so c4.5mn2 is the ceiling.

Usage

Each adapter dir bundles its tokenizer (defines the <lat_i> tokens) and — for the mn2 checkpoints — a latent_vocab.json sidecar mapping every <lat_i> back to its base-token span (needed only to expand the latent output into readable text or to retrain; plain inference does not need it). Decode in two greedy stages with a logit mask:

1. 1.Latent — from the prompt + <latent_start>, restrict next-token logits to the <lat_i> units ∪ <latent_end>; stop on <latent_end> (or a cap ~400).
2. 2.Answer — block the <lat_i> units and <latent_end>; decode <answer>X</answer>.

from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel
import torch

name = "c4.0mn2_3b_ep5"
proc = AutoProcessor.from_pretrained(f"leapeto/mindcube-latent-ckpts/{name}")
base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct", dtype=torch.bfloat16, device_map="cuda")
base.resize_token_embeddings(len(proc.tokenizer))     # absorb the added <lat_i> units
model = PeftModel.from_pretrained(base, f"leapeto/mindcube-latent-ckpts/{name}").eval()
# then the two-stage masked greedy decode above.

M-RoPE gotcha: in a manual KV-cache decode loop, pass cache_position explicitly every step. Qwen2.5-VL derives decode positions from cache_position + rope_deltas; omitting it feeds every generated token at position 0 and produces repetition/garbage (transformers.generate / vLLM handle this automatically).

Prompt format: MindCube aug_cgmap_ffr_out. Benchmark: MindCube (arXiv:2506.21458).

Full-parameter checkpoints — the ground-truth-corpus line (docs 45–54)

Everything below is full fine-tuning (ViT included) of Qwen/Qwen2.5-VL-3B-Instruct on MindCube's ground-truth map+reasoning corpus (not LoRA, not self-distilled). Each directory is a complete HF model (4 fp32 safetensors shards, ~16 GB, + processor/tokenizer): load it directly with Qwen2_5_VLForConditionalGeneration.from_pretrained(dir) — no PEFT, no `resize_token_embeddings`. Eval protocol for every number: greedy, N=1,050 tinybench, best of ep4–6.

Reference points at the same recipe and seed (checkpoints pruned, decodes in the data repo): plain +6 epochs = 70.95; latent BPE vocab cb 1.8× / 2.0× / 2.4× = 67.81 / 64.48 / 67.33. Report: docs/54_TEXT_COMPRESSION_BASELINES.md in the code repo (branch mindcube); training data under gt_corpus/text_baselines/ in leapeto/mindcube-latent-data. Recipe: warm-start from plain_s42_ep3, lr 1e-5 cosine, effective batch 512, bf16 FSDP, seed 777; on 32 GB cards use micro-batch 1 × grad-accum 128.

from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from huggingface_hub import snapshot_download
import torch
d = snapshot_download("leapeto/mindcube-latent-ckpts", allow_patterns=["text_baselines_s777/cod_ep5/*"]) + "/text_baselines_s777/cod_ep5"
proc = AutoProcessor.from_pretrained(d)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(d, dtype=torch.bfloat16, device_map="cuda").eval()
# plain greedy generate on the MindCube `plain_cgmap_ffr_out` prompt; parse the letter from <answer>…</answer>