OptGear/Opt.Gear-1B-qat-int4-per-channel-unquantized

Opt.Gear-1B-QAT

<!-- TODO: 로고 이미지 URL 교체 --> <img width="1000px" src="./OptAIxHuggingFace04.png">

  

[!Note] This repository contains the Quantization-Aware Training (QAT) checkpoint of Opt.Gear-1B: master weights adapted to the deployment INT4 grid, together with the learned channel-wise quantization scales. The deployment format is W4A16 (per-channel symmetric INT4 weights, 16-bit activations). The checkpoint performs best with its learned QAT scales, and also re-quantizes stably with other INT4 PTQ methods (RTN, AWQ, GPTQ). For the full-precision model, see Opt.Gear-1B. <!-- TODO: 링크 확인 -->

Making a model small is not enough for real devices. Beyond parameter count, the memory occupied by weights, memory bandwidth, arithmetic precision, and the kernels supported by the accelerator determine final inference performance. The question is: how much of the original quality survives when the model is squeezed into 4 bits?

Opt.Gear-1B-QAT adapts the trained model directly to the deployment INT4 grid via Quantization-Aware Training. On six English/Korean benchmarks, the QAT model loses only ~5.8 point on average from the Base model — under the exact per-channel symmetric W4A16 grid used at deployment time.

We recommend using this model if your deployment framework lacks support for group-wise PTQ or if you prioritize inference speed over accuracy; otherwise, group-wise QAT weights are preferable for maximum accuracy.

QAT Highlights

• —Group-size-32 symmetric INT4 weight quantization with a W4A16 deployment format
• —Hessian-based scale initialization: channel-wise INT4 scales initialized by a layer-wise output reconstruction objective measured on calibration activations — not simple max-based statistics
• —Learned Step Size Quantization (LSQ): weights and step sizes jointly optimized under the full language-modeling objective, with straight-through gradient estimation
• —No quantizer mismatch: the same channel-wise INT4 grid is used during initialization, QAT, and final weight conversion, so the learned quantization structure survives deployment export
• —Quantization-friendly weights: the checkpoint re-quantizes stably with RTN/AWQ/GPTQ, giving deployment pipelines freedom in quantizer choice

For more details, please refer to our tech report and blog post. <!-- TODO: 링크 연결 -->

Quantization Configuration

The scale for each quantization group is initialized as

Δ* = argmin_Δ ‖ WX − Q_Δ(W)X ‖²_F

where W is the full-precision weight matrix, X contains the calibration inputs, and Q_Δ(W) denotes the weights quantized with scale Δ. This objective accounts for the relative importance of different input directions instead of treating all weight perturbations equally. Starting from this initialization, QAT jointly refines the weights and the LSQ step sizes under the global training objective, while the forward pass consistently emulates the final W4A16 deployment format.

Benchmark Results

Base, QAT, and PTQ results of Opt.Gear-1B and Gemma3-1B on six English/Korean benchmarks. RTN, AWQ, and GPTQ rows re-quantize the master weights of the QAT checkpoint with each PTQ method; the QAT row evaluates the final QAT quantizer including learned scales. All results are obtained using the LM Evaluation Harness.

<div style="max-width:1000px;margin:0 auto;padding:16px 0"> <table style="border-collapse:collapse;font-size:15px"> <thead><tr> <th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #5b4bcf;color:#5b4bcf">Method</th> <th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">MMLU</th> <th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">HellaSwag</th> <th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">WinoGrande</th> <th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">KMMLU</th> <th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">KoBEST</th> <th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">HAERAE</th> <th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">Avg.</th> <th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">Δ</th> </tr></thead> <tbody> <tr><td colspan="9" style="padding:8px 12px;font-weight:600;color:#5b4bcf;border-bottom:1px solid rgba(91,75,207,0.2);background:rgba(91,75,207,0.08)">Opt.Gear-1B</td></tr> <tr> <td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">Base</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">43.2</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">56.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">58.2</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">36.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">60.3</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">44.2</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">49.7</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">–</td> </tr> <tr> <td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">QAT</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">37.1</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">51.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">55.9</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">30.4</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">55.4</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">33.9</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">43.9</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">-5.8</td> </tr> <tr> <td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">RTN</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.7</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.3</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">56.5</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">30.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">33.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">43.7</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">-6.0</td> </tr> <tr> <td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">GPTQ</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.3</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.7</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.3</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">25.5</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">54.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">33.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">42.7</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">-7.0</td> </tr> <tr><td colspan="9" style="padding:8px 12px;font-weight:600;color:#5b4bcf;border-bottom:1px solid rgba(91,75,207,0.2);background:rgba(91,75,207,0.08)">Gemma3-1B (reference)</td></tr> <tr> <td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">Base</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">39.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">60.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">58.7</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">30.7</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.5</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.3</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">47.3</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">–</td> </tr> <tr> <td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">QAT</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.4</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">26.9</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">56.2</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">31.2</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">42.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">-4.6</td> </tr> <tr> <td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">RTN</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">32.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">49.9</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">57.1</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">24.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">54.8</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">23.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">40.4</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">-6.9</td> </tr> <tr> <td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">GPTQ</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">31.9</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">48.1</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.2</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">24.9</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">53.1</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">26.6</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">40.0</td> <td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">-7.3</td> </tr> </tbody> </table> <p style="margin-top:12px;font-size:11px;opacity:0.7"> </p> </div>

Re-quantization Stability

The QAT checkpoint performs best with its learned scales, and the QAT-trained weights themselves respond stably to other INT4 quantizers:

Under identical RTN re-quantization, Opt.Gear-1B loses only 0.2 points on average, versus 2.4 points for Gemma3-1B. The Opt.Gear-1B-QAT checkpoint is not merely a set of learned scale parameters — the weights themselves have been adapted to be quantization-friendly, giving deployment pipelines freedom in quantizer choice.

Quickstart

[!Important] This checkpoint is intended for INT4 deployment. For best quality, export with the included learned channel-wise scales (the same per-channel symmetric W4A16 grid used during QAT). Re-quantizing with RTN/AWQ/GPTQ is supported and remains stable, at a small cost in accuracy (see table above).

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OptGear/Opt.Gear-1B-qat-int4-per-channel-unquantized" 

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

messages = [
    {"role": "user", "content": "온디바이스 AI가 왜 중요한지 설명해줘."},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

<!-- TODO: INT4 export / 디바이스 배포(QAIRT, CoreML) 가이드 링크 추가 -->

Best Practices

1. 1.Use the learned scales when possible: The learned QAT quantizer gives the best accuracy (Avg. 43.9). Switch to RTN/GPTQ only when your deployment toolchain requires it — the accuracy cost is small but non-zero.

1. 1.Keep the grid consistent: The checkpoint was trained against a per-channel symmetric INT4 grid with 16-bit activations (W4A16). Exporting to a different group size or asymmetric grid discards the learned quantization structure.

1. 1.Termination tokens: Same as the full-precision model — the Instruction model terminates with <EOT>, the Base model with <EOS>.

1. 1.Model capabilities: Task capability follows Opt.Gear-1B — Korean-English bilingual understanding and instruction following; not intended for code generation. See the base model card for full benchmark results and architecture details.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{optgear2026,
    title  = {{Opt-Gear} Technical Report},
    author = {{Opt.Gear Team}},
    year   = {2026},
    url    = {https://arxiv.org/abs/2608.01034}
}

Correspondence: contact@opt-ai.kr · Hugging Face: huggingface.co/OptGear