Modelpublic

TheHouseOfTheDude/Ornith-1.0-35B-W8A16-PTQ

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

1likes12downloads

Model Card

Ornith-1.0-35B-INT8(W8A16) — PTQ Quantized (W8A16)

Overview

Post-Training Quantized (PTQ) version of Ornith-1.0-35B.

—No calibration dataset
—One-shot quantization
—Uses llmcompressor

Quantization

—Scheme: W8A16
—Weights: INT8
—Activations: FP16/BF16
—Targets: Linear layers only

Ignored:

—lm_head
—visual
—linear_attn
—mtp

KLD Results:

—Mean KLD: 0.008166

Usage

bash

vllm serve TheHouseOfTheDude/Ornith-1.0-35B-W8A16-PTQ \
  --quantization compressed-tensors

Notes

—Requires vLLM
—Not for vanilla Transformers