Modelpublic

simaai/Qwen3-4B-Autoround-Safetensors

sourceHugging Faceapache-2.0updated 26d agoView on Hugging Face

1likes194downloads

Model Card

Qwen3-4B Autoround

Pre-LLiMa Hugging Face checkpoint based on `Qwen/Qwen3-4B` at source revision 1cfa9a7208912126459214e8b04321603b3df60c. It remains subject to the base model's license, intended use, and limitations.

Quantization

Component	Method	Weight format	Details
Decoder Linear layers	AutoRound	symmetric INT4, G256	252 targets; `lm_head` excluded
`lm_head`	GPTQ	symmetric INT4, G256	static act-order, block size 128, dampening 0.01
Mixed-precision exceptions	—	source dtype	non-Linear parameters remain at source precision

Calibration used HuggingFaceH4/ultrachat_200k (dataset revision not captured), train_sft[:512], rendered with the source chat template. Token IDs were deterministically concatenated and packed into 512 full 1024-token spans (no shuffle); AutoRound used 200 iterations and batch size 1.

Evaluation

Matched full WikiText evaluation used the wikitext task backed by EleutherAI/wikitext_document_level, wikitext-2-raw-v1, with no example limit and the same evaluator settings for source and quantized checkpoints.

Checkpoint	Word perplexity
Source (`Qwen/Qwen3-4B`)	18.454564
This UltraChat-calibrated checkpoint	20.196137

The source-to-quantized change is +1.741573 PPL (+9.437%). quantize.py completed finite-scale validation and a Transformers chat-generation smoke test.

Reproduction

This directory includes the exact quantize.py, recipe.yaml, and versions.txt.

bash

python quantize.py --model-path /path/to/models--Qwen--Qwen3-4B --output-dir /path/to/output

Environment

Exact Python, CUDA, Torch, Transformers, llmcompressor, AutoRound, and compressed-tensors versions are recorded in versions.txt.

Deployment

This is the pre-LLiMa quantized Hugging Face artifact. Compile it separately for the target Sima.ai platform and keep compiler output separate.

Limitations

Quantization quality can vary by language, domain, prompt format, context length, and deployment runtime; validate the intended workload independently.