Modelpublic

JC1DA/Qwen3.8-27B-heretic-ara-W4A16

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes1.2kdownloads

Model Card

JC1DA/Qwen3.8-27B-heretic-ara-W4A16

4-bit weight quantized version of trohrbaugh/Qwen3.8-27B-heretic-ara using AutoRound.

Quantization Details

—Scheme: W4A16 (4-bit weights, 16-bit activations)
—Format: auto_round
—Library: AutoRound
—Iterations: 1000
—Torch compile: enabled
—Group size: 128
—Ignored layers: embedtokens, visual layers, mtp layers, inputlayernorm, postattentionlayernorm, qnorm, knorm, conv1d, linear_attn.norm

Original quantization command

bash

auto-round --model trohrbaugh/Qwen3.8-27B-heretic-ara --scheme "W4A16" --format "auto_round" --output_dir ./Qwen_Qwen3.8-27B-INT4-W4A16-Autoround --iters 1000 --enable_torch_compile --ignore_layers "model.language_model.embed_tokens,model.visual.*,mtp.*,input_layernorm,post_attention_layernorm,q_norm,k_norm,conv1d,linear_attn.norm"

Usage

Load with AutoRound's loader:

python

from autoround.autoround_utils import load_model
model = load_model(model_path="JC1DA/Qwen3.8-27B-heretic-ara-W4A16")

Or with standard transformers (if compatible with your setup):

python

from transformers import AutoModelForCausalLM, AutoProcessor
model = AutoModelForCausalLM.from_pretrained("JC1DA/Qwen3.8-27B-heretic-ara-W4A16", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("JC1DA/Qwen3.8-27B-heretic-ara-W4A16")

Model Info

—Parameters: ~27B
—Sharded: Yes (10 shards)
—Files: model-00001-of-00010.safetensors through model-00010-of-00010.safetensors