Modelpublic

darthcrawl/EVA-Qwen2.5-32B-v0.2-mlx-8bit

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes32downloads

Model Card

EVA-Qwen2.5-32B-v0.2-mlx-8bit

MLX 8-bit quant of EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2. Disk: ~32 GB. Active memory: ~34 GB plus KV cache.

8-bit is statistically indistinguishable from BF16 here. Pick this if you want max EVA quality and have memory; the 4-bit is fine for most uses.

Quantization

bash

mlx_lm.convert --hf-path EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2 \
  --mlx-path ./EVA-Qwen2.5-32B-v0.2-mlx-8bit -q --q-bits 8

REQUIRED: EOS patch before serving

Same as the 4 and 6-bit. The source generation_config.json is missing <|im_end|> from eos_token_id. Patch:

bash

python3 -c "
import json
p = './EVA-Qwen2.5-32B-v0.2-mlx-8bit/generation_config.json'
g = json.load(open(p))
g['eos_token_id'] = [151643, 151645]
json.dump(g, open(p, 'w'), indent=2)
"

This quant has it applied.

Recommended sampling

Same as 4 and 6-bit:

yaml

temperature: 1.0
min_p: 0.05
repetition_penalty: 1.05
repetition_context_size: 2048

Known mlx_lm.server bugs

Same as the smaller quants (XTC tokenizer bug, seed ignored, prompt-cache size cap recommended).

License

Apache-2.0 from base.