pyloxsystems/pylox-voice-8b

Pylox Voice 8B — Llama-3.1-8B LoRA (SFT + EAGLE-3)

Model: pyloxsystems/pylox-voice-8b Base model: meta-llama/Llama-3.1-8B-Instruct Method: Supervised Fine-Tuning (SFT) + EAGLE-3 speculative decoding at serve time Domain: Multi-turn conversational dialogue for voice-AI applications Hardware: NVIDIA Grace Blackwell GB10 (DGX Spark, 128 GB unified memory)

Abstract

We present a parameter-efficient LoRA adapter fine-tuned on the Capybara multi-turn conversation dataset for the task of natural, flowing conversational dialogue generation — optimized for voice-AI latency budgets. Starting from meta-llama/Llama-3.1-8B-Instruct, we apply QLoRA (NF4, rank 32) across all seven linear projection layers via three epochs of supervised fine-tuning on 15,204 curated multi-turn conversation examples drawn from the full LDJnr/Capybara corpus. The adapter shifts the base model's register toward natural, multi-turn spoken-language dialogue patterns. Deployed with the RedHatAI EAGLE-3 speculative head, it achieves 2–3× lower first-token latency — critical for sub-300ms voice agent response budgets. MT-Bench overall score: 5.875 / 10 (target 6.5; humanities 7.41, STEM 6.77, writing 6.83 are strong; math and reasoning reflect the 8B model's inherent ceiling). Serves at 34+ tok/s on a single NVIDIA Grace Blackwell GB10.

Model Details

Intended Use

Primary use cases

• —Natural multi-turn dialogue generation for voice-AI assistants and agents
• —Conversational response generation where naturalness and turn coherence matter
• —Low-latency inference for real-time voice applications (Vapi, Retell, Bland, Voiceflow)
• —Avatar and video-gen conversational backends (Tavus, Argil, Synthesia)
• —Peer support and wellness conversational agents

Target users

Voice AI platform engineers, conversational AI developers, and researchers building real-time dialogue systems. Available under the Llama 3.1 Community License.

Out of scope

Not designed for single-turn Q&A, retrieval-augmented generation, or structured output tasks. The adapter improves multi-turn conversational register — it does not add factual knowledge or tool-calling capability. Not evaluated for clinical or crisis-support applications.

Training Data

Preprocessing methodology

1. 1.Format normalization — Capybara conversation: [{input, output}] structure converted to standard messages format (user/assistant alternating turns)
2. 2.Near-duplicate removal — MinHash with Jaccard similarity threshold 0.92
3. 3.Quality scoring — Each example scored 1–5 by a local 120B judge on conversational naturalness and turn coherence; examples below 3/5 discarded
4. 4.Full corpus — No row cap applied; all 16,006 source examples processed (per pipeline policy: never subsample training data)

Training Procedure

SFT Hyperparameters

Infrastructure

Evaluation

MT-Bench

Evaluated via single-grade MT-Bench (80 questions, 2 turns each) using openai/gpt-oss-120b as judge (local TRT-LLM endpoint). Date: 2026-05-01.

The target was ≥6.5 overall. The adapter scores 5.875 — strong on the categories most relevant to voice applications (humanities, writing, roleplay) and weaker on math and reasoning, which is expected for an 8B SFT adapter without RL alignment on those domains.

Safety

Red-team evaluated against a 50-prompt adversarial suite (JailbreakBench, AdvBench, PAIR, DAN archive).

Quickstart

PEFT (direct adapter loading)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_id = "meta-llama/Llama-3.1-8B-Instruct"
adapter_id = "pyloxsystems/pylox-voice-8b"

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(
    base_id, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_id)

messages = [
    {"role": "user", "content": "I've been feeling really disconnected lately. Like I'm going through the motions but nothing feels real."},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
print(tokenizer.decode(out[0], skip_special_tokens=True))

vLLM with EAGLE-3 (low-latency serving)

vllm serve nvidia/Llama-3.1-8B-Instruct-NVFP4 \
    --enable-lora \
    --lora-modules pylox-voice-8b=pyloxsystems/pylox-voice-8b \
    --speculative-config '{"method": "eagle3", "model": "RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3", "num_speculative_tokens": 5}'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
response = client.chat.completions.create(
    model="pylox-voice-8b",
    messages=[
        {"role": "system", "content": "You are a warm, natural conversational assistant."},
        {"role": "user", "content": "I've been really stressed about this job interview tomorrow."},
    ],
    max_tokens=256,
    temperature=0.7,
)
print(response.choices[0].message.content)

Limitations

• —MT-Bench 5.875 vs 6.5 target: Math (5.5) and reasoning (4.33) categories reflect the 8B model's inherent capability ceiling on multi-step reasoning, not a training failure. For voice applications (writing, roleplay, humanities), the scores are strong.
• —Conversational register only: The adapter improves multi-turn naturalness, not factual accuracy or domain knowledge. Inject domain knowledge via system prompt or RAG.
• —English only: Not evaluated on multilingual dialogue.
• —Max sequence length 2,048: Long conversation histories must be truncated or summarized.

Citation

@misc{pylox_voice_8b_2026,
  author       = {Girard, Emilio},
  title        = {Pylox Voice 8B -- Llama-3.1-8B LoRA for Voice AI},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/pyloxsystems/pylox-voice-8b}}
}

Built at Pylox Forge — on-prem LLM fine-tuning and deployment on NVIDIA Grace Blackwell hardware.