Modelpublic

darthcrawl/anubis-70b-v1.2-mlx-6bit

sourceHugging Facellama3.1updated 5mo agoView on Hugging Face

0likes57downloads

Model Card

Anubis-70B-v1.2-mlx-6bit

MLX 6-bit quant of TheDrummer/Anubis-70B-v1.2. Llama-3.1-70B base, RP finetune. Disk: ~53 GB. Active memory while serving: ~55 GB plus KV cache.

Quality is closer to BF16 than 4-bit at the cost of disk and decode speed. Pick this if you have memory headroom on your machine and want sharper outputs.

Quantization

bash

mlx_lm.convert --hf-path TheDrummer/Anubis-70B-v1.2 \
  --mlx-path ./anubis-70b-v1.2-mlx-6bit -q --q-bits 6

Recommended sampling (RP)

yaml

temperature: 0.95
min_p: 0.05
repetition_penalty: 1.05
repetition_context_size: 2048

Speculative decoding

Same draft as the 4-bit (mlx-community/Llama-3.2-3B-Instruct-uncensored-6bit). --num-draft-tokens 7.

Known mlx_lm.server bugs

Same as the 4-bit:

—seed parameter silently ignored in 0.31.3.
—Prompt-cache KV grows unbounded; set --prompt-cache-size 6 or lower at this size.

License

Inherits llama3.1 from base.