darthcrawl/artifex-rp-orpheus-llama-3.1-8b-gguf

artifex-rp-orpheus-llama-3.1-8b — GGUF (register-calibrated)

Forging voices for the machines people actually own.

GGUF quants of darthcrawl/artifex-rp-orpheus-llama-3.1-8b for llama.cpp / Ollama / LM Studio / text-generation-webui. Built for CPU and mid-range-GPU inference on commodity desktops.

Part of the [Artifex RP](https://huggingface.co/collections/darthcrawl) series — register-craft models tiered to common hardware. Curated small data, not scraped giant data. Anti-corporate-assistant by design.

Register: restrained-lyrical — vivid, intelligent prose; tension carries more weight than description.

[image]

"He meant the sentence differently than he knew how to say."

For the full model card see the base repo.

Pick a quant

These are register-calibrated (-register) — the importance matrix was built on Orpheus's own prose, not generic wikitext, so the low-bit quants preserve the voice, not just generic perplexity. kld is the mean KL-divergence vs the F16 model on held-out register text (lower = closer to F16). Pick the largest that fits your RAM/VRAM.

Sweet spot for most users is Q4_K_M. (Q8_0 needs no imatrix, so there's one file; everything else is register-calibrated.)

Why register-calibrated, and does it help?

The importance matrix (imatrix) that guides quantization was computed on Orpheus's own register (its training prose) rather than the usual generic wikitext. The quants are then scored by KL-divergence vs F16 on held-out register text — the metric that actually reflects use-case quality, where wikitext perplexity hides register degradation.

Each quant was built both ways and measured. Register-calibrated wins at every size, and by the largest margin at the lowest bit-rate, where quantization damage is worst:

(Q8_0 anchor KLD 0.00090; F16 = reference. The raw per-quant measurements are in [`results.csv`](./results.csv).)

Takeaway: at the bit-rates people on commodity hardware actually run, calibrating on the model's own voice measurably preserves more of it.

Run it

llama.cpp

./llama-cli -m orpheus-Q4_K_M-register.gguf \
  --jinja \
  -p "She hadn't seen him in three years. He called on a Tuesday." \
  -n 400 --temp 0.85 --top-p 0.95 --repeat-penalty 1.1

Ollama

FROM ./orpheus-Q4_K_M-register.gguf

PARAMETER temperature 0.85
PARAMETER top_p 0.95
PARAMETER repeat_penalty 1.1

SYSTEM """Write in vivid, restrained, intelligent prose. Concrete sensory detail. Never pretentious. Tension carries more weight than description. Match the user's energy and length."""

Then ollama create artifex -f Modelfile && ollama run artifex.

LM Studio / Jan / text-generation-webui

Drop the .gguf into the loader's model directory, pick Q4_K_M-register unless your RAM/VRAM dictates otherwise. Temp 0.85, top-p 0.95, repetition penalty 1.1. Llama-3 chat template (<|begin_of_text|> / <|eot_id|>).

System prompt

Write in vivid, restrained, intelligent prose. Concrete sensory detail.
Never pretentious. Tension carries more weight than description.
Match the user's energy and length.

License

Llama 3.1 Community License (inherited from base). See LICENSE.