CoolFace
Datasetpublic

Venugopalan2610/trellis-vs-scalar-2bit-qat-snapshots

Single-Layer 2-Bit QAT Weight Snapshots Per-layer 2-bit quantized weight snapshots for the controlled trellis-vs-scalar comparison in "Small GPU, Small Edge: A Controlled Study of Trellis vs. Scalar 2-Bit QAT." These reproduce Tables 1, 3, and 4 of the paper. Built on meta-llama/Llama-3.2-1B and subject to the Llama 3.2 Community License. Each snapshot quantizes a single decoder layer's 7 projections to 2 bits; all other layers stay FP16. These are experiment artifacts, not a… See the full description on the dataset page: https://huggingface.co/datasets/Venugopalan2610/trellis-vs-scalar-2bit-qat-snapshots.

sourceHugging Facellama3.2updated 3mo agoView on Hugging Face
0likes16downloads
Dataset Card

Single-Layer 2-Bit QAT Weight Snapshots

Per-layer 2-bit quantized weight snapshots for the controlled trellis-vs-scalar comparison in "Small GPU, Small Edge: A Controlled Study of Trellis vs. Scalar 2-Bit QAT." These reproduce Tables 1, 3, and 4 of the paper.

Built on meta-llama/Llama-3.2-1B and subject to the Llama 3.2 Community License. Each snapshot quantizes a single decoder layer's 7 projections to 2 bits; all other layers stay FP16. These are experiment artifacts, not a deployable model.

Contents

filelayerarminit → train
snapshots/layer04_bcjr_seed{0,1,2}.pt4 (mild)BCJR-trellis (soft)PTQ → 10-step BCJR
snapshots/layer04_ste_trellis_seed0.pt4STE-trellisFP → 10-step STE
snapshots/layer04_scalar_ctrl_seed0.pt4scalar-ctrl (uniform)FP → 10-step STE
snapshots/layer01_bcjr_seed0.pt1 (sensitive)BCJR-trellisPTQ → 10-step BCJR
snapshots/layer01_ste_trellis_seed0.pt1STE-trellisFP → 10-step STE
snapshots/layer01_scalar_ctrl_seed0.pt1scalar-ctrlFP → 10-step STE

The trellis-PTQ and scalar-PTQ arms are not included — they carry no training and are recomputed on the fly from the FP weights.

Format

Each .pt is a torch.save dict {proj_name: W_q} (fp32/bf16, original un-rotated basis), with keys attn_{q,k,v,o}_proj and mlp_{gate,up,down}_proj.

Usage (reproduce the tables)

python
import torch
from transformers import AutoModelForCausalLM
from src.qat.eval_llama_layer import install_layer_weights   # from the code repo

m = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B", torch_dtype=torch.bfloat16, device_map="cuda")
snap = torch.load("snapshots/layer04_bcjr_seed0.pt", weights_only=True)
install_layer_weights(m, 4, snap, dtype=torch.bfloat16)
# then eval WikiText-2 PPL / lm-eval 0-shot -> Tables 1, 3, 4
  • —Code: https://github.com/Venugopalan2610/quant-2bit
  • —Paper: Small GPU, Small Edge (arXiv link TBD)