ssurface/cot-baseline-tokenskip-g70

TokenSkip baseline (g70) — our reimplementation

TokenSkip-style compression trained on our base and evaluated in our harness, so the comparison against the dialect adapters is like-for-like rather than a number copied across papers.

Results

GSM8K test (n=1317), greedy decoding, single-turn, no exemplars, no self-consistency.

Training data

Training setup

Loss is on the completion only, with prompt lengths precomputed at load time rather than found by pattern search — the pattern-search collator silently masked nothing, which let the base model's tool-calling prior leak into the chains.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "ssurface/cot-baseline-tokenskip-g70")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")

Limitations

• —Trained and evaluated on math word problems only.
• —Accuracy falls with problem difficulty, fastest at the compressed levels.
• —Single seed unless the repo name says otherwise; differences of a couple of points are within noise (95% half-width ~2.7 pp at n=1317, ~4.4 pp at n=500).

Citation

@misc{cot-compression-dialects,
  title  = {Chain-of-Thought Compression Dialects},
  author = {Frolov, Anatolii},
  year   = {2026}
}