emiliogirard/cs-bitext-llama-3.1-8b-lora-dpo

CS Bitext — Llama-3.1-8B LoRA (SFT + DPO)

Model: emiliogirard/cs-bitext-llama-3.1-8b-lora-dpo Base model: meta-llama/Llama-3.1-8B-Instruct Method: Supervised Fine-Tuning (SFT) + Direct Preference Optimization (DPO) Domain: Customer support dialogue generation Hardware: NVIDIA Grace Blackwell GB10 (DGX Spark, 128 GB unified memory)

Abstract

We present a parameter-efficient LoRA adapter fine-tuned on the Bitext customer support LLM chatbot training dataset for the task of intent-aware customer support response generation. Starting from meta-llama/Llama-3.1-8B-Instruct, we apply QLoRA (NF4, rank 32) across all seven linear projection layers via three epochs of supervised fine-tuning on 378 curated support dialogue turns, followed by one epoch of DPO alignment on 132 preference pairs. The adapter learns the structural conventions, tone, and intent-handling patterns of professional customer support dialogue — including billing inquiries, order status, account management, product information, and complaint resolution — without encoding product-specific knowledge. The SFT checkpoint achieves a 32% pairwise win rate over the base model on held-out support prompts (scored by a local 120B judge). It serves at 27.9 tok/s on a single NVIDIA Grace Blackwell GB10 with NVFP4 quantization and EAGLE-3 speculative decoding, approximately 20× cheaper per token than GPT-4o for equivalent support dialogue generation.

Model Details

Intended Use

Primary use cases

This adapter is intended for customer support dialogue generation tasks. Suitable applications include:

• —Generating contextually appropriate, on-topic responses to customer inquiries
• —Handling intent categories including billing, account management, order status, product information, and complaint resolution
• —Powering customer-facing chatbot products where domain-specific knowledge is supplied via system prompt or RAG
• —Rapid prototyping of support automation pipelines

Target users

Developers building customer-facing chatbot products, support automation engineers, and researchers working on task-oriented dialogue systems. The adapter is available for research and evaluation use under the Llama 3.1 Community License.

Out of scope

This adapter is a domain-style adapter — it learns conversational structure and tone, not product-specific knowledge. It does not encode knowledge about any particular company's products, policies, or account systems. It should not be used as a replacement for human agents in high-stakes support scenarios requiring legal judgment, escalation handling, or accessibility accommodations. It is not suitable for healthcare or financial services contexts where regulatory compliance governs customer communications.

Training Data

Preprocessing methodology

1. 1.Near-duplicate removal — MinHash with Jaccard similarity threshold 0.92; removes near-identical dialogue turns while preserving intent-category diversity
2. 2.Quality scoring — Each example is scored 1–5 by a local 120B judge model evaluating response helpfulness, appropriateness of tone, intent resolution quality, and absence of hallucinated policy claims; examples scoring below 3/5 are discarded
3. 3.PII redaction — Named entity recognition pass removes customer names, order numbers, account identifiers, and addresses from examples
4. 4.DPO pair construction — 132 preference pairs generated by sampling two responses per prompt from the SFT model (temperature 0.7 and 0.0), then ranked by the 120B judge on helpfulness and tone; used for the DPO alignment pass

Training Procedure

SFT Hyperparameters

DPO Hyperparameters

Infrastructure

Evaluation

Inference Performance

Measured against a live vLLM endpoint (NVFP4 + EAGLE-3, LoRA hot-loaded) on NVIDIA Grace Blackwell GB10. Target response length: 150 tokens.

Cost Analysis

Self-hosted electricity cost at $0.05/hr (Montreal hydro). Compute cost approaches $0 once hardware is amortized.

Academic Benchmarks

Evaluated via lm-evaluation-harness 0.4.x (local-completions model class) against the live vLLM NVFP4+EAGLE-3 endpoint. Tokenizer: nvidia/Llama-3.1-8B-Instruct-NVFP4. Limit: 50 samples per subtask. Date: 2026-05-01.

Domain Benchmarks

No standard off-shelf lm-evaluation-harness benchmark directly covers customer support dialogue quality.

LLM Judge (Pairwise Win Rate)

Pairwise comparison scored by a local gpt-oss-120b TRT-LLM judge. The judge receives a prompt plus two responses (finetune vs base model, order randomized) and picks the better one. Base model: meta-llama/Llama-3.1-8B-Instruct loaded in NF4 via bitsandbytes + PEFT. Evaluated on 50 held-out prompts from formatted_eval.jsonl. Date: 2026-05-01.

Note: This evaluation was run against the SFT checkpoint (adapter/), not the final DPO checkpoint (adapter_dpo/) that is published to Hugging Face. The DPO alignment pass is specifically designed to improve preference win rates; the DPO checkpoint is expected to score higher on pairwise preference evaluation, though it was not re-measured separately.

The SFT adapter wins on 32% of pairwise comparisons. This reflects the adapter's learned structure and tone conventions (intent acknowledgment, resolution confirmation, professional register) that the base model's general-purpose responses lack, while the base model retains an advantage on prompts where general helpfulness outweighs support-specific format.

Safety

Red-Team Evaluation

Evaluated against a 50-prompt adversarial suite drawn from JailbreakBench, AdvBench, PAIR, and the DAN archive. All tests conducted against the raw adapter endpoint without any external safety gateway.

The adapter inherits the safety alignment of the base Llama-3.1-8B-Instruct model. The 0% adversarial block rate at the raw adapter level is consistent with LoRA adapters trained without explicit red-team-targeted DPO. A 3-layer safety gateway (regex shields → Meta Prompt Guard 2 → Meta Llama Guard 3) is available via pylox deploy --with-safety. Customer-facing deployments should enable the gateway.

Limitations

• —No product-specific knowledge: The adapter captures support dialogue style and structure, not any company's policies, products, or pricing. Product-specific context must be injected via system prompt or retrieval-augmented generation.
• —Dataset scope: Trained on 378 examples from the Bitext dataset, which represents generic English-language B2C support conventions. May not generalize to B2B, enterprise SLA, or highly regulated industry support contexts.
• —English only: Not evaluated on multilingual support scenarios.
• —SFT win rate only: LLM judge win rate (32%) was measured on the SFT checkpoint; DPO checkpoint win rate was not separately measured.
• —Sequence length: Max\seq\length=2,048. Long conversation histories must be truncated or summarized.

Bias, Fairness, and Ethical Considerations

This adapter produces responses based on patterns learned from Bitext customer support examples, which reflect generic English-language B2C support conventions. It may produce responses that assume Western business norms, return policies, or communication styles not applicable to the deployer's product or customer base. Outputs should be reviewed for brand fit and regulatory compliance before deployment. The adapter must not be used to handle sensitive customer data, financial disputes, or legal escalations without human oversight.

Quickstart

PEFT (direct adapter loading)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_id = "meta-llama/Llama-3.1-8B-Instruct"
adapter_id = "emiliogirard/cs-bitext-llama-3.1-8b-lora-dpo"

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(
    base_id, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_id)

prompt = (
    "You are a helpful customer support assistant.\n\n"
    "Customer: I placed an order three days ago and I haven't received a shipping "
    "confirmation yet. Order number is #84721. Can you help me?\n\n"
    "Support Agent:"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

vLLM (OpenAI-compatible API)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8010/v1", api_key="none")
response = client.chat.completions.create(
    model="cs-bitext",  # vLLM LoRA mount name
    messages=[
        {"role": "system", "content": "You are a helpful customer support assistant."},
        {"role": "user", "content": "I was charged twice for my last order. How do I get a refund?"}
    ],
    max_tokens=256,
)
print(response.choices[0].message.content)

Citation

@misc{girard_cs_bitext_2026,
  author       = {Girard, Emilio},
  title        = {CS Bitext -- Llama-3.1-8B LoRA (SFT + DPO)},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/emiliogirard/cs-bitext-llama-3.1-8b-lora-dpo}}
}

Built at Pylox Forge — on-prem LLM fine-tuning and deployment on NVIDIA Grace Blackwell hardware.