Modelpublic

sotanengel/Qwen2.5-Coder-32B-Instruct-GPTQ-Int4

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes11downloads

Model Card

Qwen2.5-Coder-32B-Instruct-GPTQ-Int4

Overview

This is a quantized version of Qwen/Qwen2.5-Coder-32B-Instruct using GPTQ with Quantization Error Propagation (QEP).

Quantization Details

Item	Value
Method	GPTQ
Bits	4-bit (wbits=4)
Group size	128
QEP	✅ Enabled
Calibration samples	512
Calibration max length	2048
Original size	~64 GB (BF16)
Quantized size	~18 GB

QEP (Quantization Error Propagation) compensates quantization errors by propagating correction terms to subsequent layers, resulting in higher accuracy than standard GPTQ.

Hardware Requirements

GPU	VRAM	Status
NVIDIA L4	24 GB	✅ OK (gptqmodel + Marlin kernel required)
NVIDIA L40S	48 GB	✅ Recommended
NVIDIA A100 40G	40 GB	✅ OK
NVIDIA A100 80G	80 GB	✅ OK

Note: L4 (24 GB) has been verified to run this model when using gptqmodel with the Marlin kernel. The transformers + standard GPTQ backend may be insufficient due to peak VRAM usage.

Benchmark Results (Coding Tasks)

Evaluated on 6 coding prompts (2 basic / 2 intermediate / 2 advanced) using gptqmodel + Marlin kernel on NVIDIA L4 (24 GB).

Performance Summary

Metric	Value
Success rate	6 / 6
Device	All GPU
Total inference time	~249 s
Per-prompt latency	~32–55 s
Generation settings	`max_new_tokens=1024`, `temperature=0.1`

Task-level Quality

#	Task	Difficulty	Result	Note
1	Quicksort	Basic	✅ Good	Type hints, docstring, correct output
2	LRU Cache	Intermediate	⚠️ Acceptable	O(1) via `OrderedDict`; custom doubly-linked list not implemented
3	2Sum	Basic	✅ Good	O(n), duplicate-pair normalization handled correctly
4	Retry Decorator	Intermediate	✅ Good	`functools.wraps`, spec-compliant
5	Binary Search Bug Fix	Advanced	❌ Needs review	`//` and `left=mid+1` are correct, but misidentified `right=mid-1` as not needing a fix for the half-open interval `[left, right)`
6	`task_queue` package	Advanced (multi-file)	✅ Good	4-file structure, re-exports per spec

Overall Assessment

Axis	Score
Correctness	7 / 10 — boundary-condition errors (esp. task 5)
Readability / Explanation	8 / 10
Spec faithfulness	6 / 10 — stdlib shortcut tendency
Speed on L4	7 / 10

Summary: Sufficient for draft and boilerplate generation. Algorithm fixes and strict correctness require human review or automated tests.

Runtime Notes

—Marlin JIT compilation requires CUDA_HOME and the CUDA toolkit to be present on the first run.
—CPU fallback is disabled (fail-fast) — a GPU is required.

Usage

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "sotanengel/Qwen2.5-Coder-32B-Instruct-GPTQ-Int4",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "sotanengel/Qwen2.5-Coder-32B-Instruct-GPTQ-Int4"
)

messages = [{"role": "user", "content": "Write a quicksort in Python."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

gptqmodel + Marlin (recommended for L4)

python

from gptqmodel import GPTQModel

model = GPTQModel.load(
    "sotanengel/Qwen2.5-Coder-32B-Instruct-GPTQ-Int4",
    device="cuda:0",
)

Deployment Options

—vLLM: OpenAI-compatible API server
—SGLang: High-performance LLM serving
—Docker Model Runner: Containerized deployment

License

Apache 2.0 (same as original model)

Credits

—Original Model: Qwen/Qwen2.5-Coder-32B-Instruct by Qwen Team / Alibaba Cloud
—Quantization: OneCompression by Fujitsu Ltd.