abhimittal/veritas-hallucination-reduction

🔬 VERITAS — Hallucination-Reduction RAG

VERITAS (Verification-Enhanced Retrieval with Iterative Truth Assessment and Scoring) is a RAG pipeline that treats hallucination as something to gate, verify, repair, and measure — not just hope away. It runs on top of any LLM (Claude, GPT, Llama, or the bundled offline mock) with zero heavy dependencies: retrieval and verification are pure Python.

The technique

Why it works: most RAG hallucinations are answers to questions the corpus can't support (killed by the gate); the rest are fabricated details woven into otherwise-grounded text (caught claim-by-claim by the dual verifier — a persuasive LLM cannot sweet-talk the lexical judge, and fabricated numbers light it up instantly).

Benchmark results

40 questions (20 answerable, 12 unanswerable, 8 adversarial) over a bundled 10-document corpus. Both systems run on the same deterministic MockLLM configured to hallucinate on 35% of its answers — simulating an unreliable base model — and are graded by the same model-free lexical judge against the full corpus. Reproduce: python benchmarks/run_benchmark.py.

VERITAS removes 100% of fabricated claims from delivered answers and cuts question-level hallucination by ~70%, with no loss of accuracy or coverage on answerable questions. (Residual "hallucinations" are strictly- scored cases where the system answered an adversarial question with true-but- irrelevant corpus facts.) Benchmark a live model with python benchmarks/run_benchmark.py --provider anthropic|openai|hf.

Technique zoo — quantitative comparison

Beyond VERITAS, the repo implements a spread of hallucination-reduction techniques from the recent literature behind one comparable interface (veritas.techniques), and benchmarks them head-to-head on the same corpus with the same independent judge. Run python benchmarks/run_comparison.py.

Representative result (40 questions, MockLLM with 35% injected hallucinations, same lexical judge for all — benchmarks/comparison.md):

(Every technique faces an identical fresh mock — same injected hallucinations — and the runner meters LLM calls per question, so cost is a first-class axis.)

The comparison is deliberately honest about tradeoffs: Semantic Entropy buys the lowest hallucination rate by abstaining aggressively (35% false-abstention, reduced coverage) at 6 calls/question; the verify family (VERITAS / Quote Grounding / Multi-Agent) hits the quality sweet spot (zero fabricated claims delivered); Guardrails shows why rails alone aren't enough (they check form, not truth); and the VERITAS Cascade matches VERITAS on every quality metric at 1.05 calls/question — a 3× cost cut with zero quality loss — by escalating only when a free lexical screen finds something shaky. There is no single winner on all axes — that's the point.

Novel features beyond the comparison

• —VERITAS Cascade (techniques/cascade.py) — cost-aware escalation: retrieval gate (0 calls) → draft (1) → free lexical screen → full dual-judge verification only for shaky drafts → semantic-entropy arbiter only after repairs. Fabricated numbers can't pass the free screen, so the fast path is safe for the failure class that matters most.
• —Evidence-conflict detection (verification.detect_evidence_conflict) — when two retrieved sources disagree about a number (8849 m vs 8850 m), standard RAG silently picks one; VERITAS flags the conflict on the answer (or abstains, via PipelineConfig(on_conflict="abstain")).
• —Hallucination stress-test generator (veritas/stresstest.py) — turns any corpus into a labeled fabrication-detection benchmark (verbatim / number-corrupted / entity-swapped / fabricated probes) and scores detectors per corruption type. It exposed a real blind spot — entity swaps evade lexical entailment completely (0% detection) — and drove the fix: an entity-consistency judge that closes the gap to 100% with no false flags. Committed results: benchmarks/stress_report.md; try your own docs in the demo's Stress test tab.
• —Trust cards (VeritasResult.to_trust_card()) — machine-readable provenance JSON per answer: claims with verdicts and evidence, citations, groundedness, repairs, and any evidence conflicts — so downstream apps can gate on the answer's support instead of trusting prose.

Run the comparison on a real model — free, on a Colab GPU

No API keys or credits needed: run any open-weights instruct model on Colab's free T4 and point the whole benchmark at it. `notebooks/colab_live_benchmark.ipynb` does it end-to-end (defaults to Qwen/Qwen2.5-7B-Instruct, 4-bit, which fits a T4). Locally with a GPU:

pip install -e ".[local]"
python benchmarks/run_comparison.py --provider local \
    --model Qwen/Qwen2.5-7B-Instruct --load-4bit --balanced 5 --out comparison_live

--provider local uses veritas.local.LocalChatClient (applies the model's chat template + optional 4-bit quantization). Paid alternatives: --provider openai --model gpt-4o-mini (~15¢) or --provider anthropic|hf.

White-box (DoLa) is compared separately because it needs logit access (python benchmarks/run_dola.py, requires pip install 'veritas-rag[local]'). A real run on gpt2 (benchmarks/dola.md): DoLa lifts answer accuracy 0% → 12.5% and groundedness 2% → 12% over vanilla greedy decoding on the same model, at ~2× latency — the expected direction (DoLa's full effect needs a larger instruction-tuned model on a factuality benchmark; gpt2 is a wiring check).

Quickstart

# pip install -e .            (pure stdlib; extras: [anthropic], [openai], [hf], [demo])
from veritas import Document, HybridRetriever, MockLLM, VeritasPipeline, chunk_corpus

docs = [Document("d1", "Mount Everest is the highest mountain on Earth. "
                       "Its summit stands at 8849 meters above sea level.")]
retriever = HybridRetriever(chunk_corpus(docs))

llm = MockLLM()                       # offline demo model
# from veritas import AnthropicClient; llm = AnthropicClient()   # or any real LLM

result = VeritasPipeline(llm, retriever).answer("How tall is Mount Everest?")
print(result.answer)          # "Its summit stands at 8849 meters above sea level. [c1]"
print(result.groundedness)    # 1.0
print(result.abstained)       # False — and True (with reason) for unanswerable questions
for verdict in result.final_verdicts:
    print(verdict.label, verdict.claim.text)

Demo

The Gradio demo (this Space) runs keyless out of the box on the deterministic mock model — flip the provider dropdown to Anthropic / OpenAI-compatible / Hugging Face Inference and paste your own API key (used per-request, never stored) to drive it with a real LLM. It shows the VERITAS answer next to the baseline RAG answer, per-claim verdicts, the full pipeline trace, and the benchmark charts.

Run locally: pip install -r requirements.txt && python app.py

Deploy your own Space: python scripts/deploy_space.py --repo <user>/veritas-demo (needs a Hugging Face write token via --token or the HF_TOKEN env var).

Repository layout

src/veritas/          the pipeline (chunking, retrieval, llm adapters, prompts,
                      claims, verification, pipeline, metrics, graph)
src/veritas/techniques/
                      the technique zoo behind one interface: semantic_entropy,
                      quote_grounding, multi_agent, guardrails, calibration,
                      graph_rag, decoding (DoLa), nli, wrappers
tests/                95 offline tests — pytest
benchmarks/           corpus + 40-question dataset + runners (run_benchmark,
                      run_comparison, run_dola) + committed results
skills/hallucination-reduction/SKILL.md
                      reusable playbook: prompting, chain-of-verification,
                      temperature settings, RAG design for ANY LLM
app.py                Gradio demo (Hugging Face Spaces entrypoint)
scripts/deploy_space.py
                      one-command Space deployment

The skill

`skills/hallucination-reduction/SKILL.md` distills the technique into a provider-agnostic playbook — grounding-contract prompting, chain-of-thought vs chain-of-verification, a per-task temperature table (including models that reject sampling params), and the RAG design checklist. Drop the skills/ folder into a Claude Code project (or any agent-skills-compatible harness) and it activates whenever you work on hallucination-sensitive LLM features.

Testing

pip install -e ".[dev]"
pytest        # 95 tests; add [local] (torch+transformers) to run the DoLa test

License

MIT