Modelpublic

llm-semantic-router/modernbert-base-32k-haldetect-combined

sourceHugging Faceapache-2.0updated 9mo agoView on Hugging Face

1likes22downloads

Model Card

🥬 ModernBERT-base-32k Hallucination Detector (Combined)

A hallucination detection model fine-tuned on RAGTruth + HaluEval datasets using extended 32K context ModernBERT.

Model Description

This model detects hallucinations in LLM-generated text by classifying each token as either Supported (grounded in context) or Hallucinated (not supported by context).

Key Features

—32K Context Window: Built on `llm-semantic-router/modernbert-base-32k` with YaRN RoPE scaling
—Multi-Dataset Training: Trained on RAGTruth (~13.5K) + HaluEval (~38K) = 48K+ samples
—Token-Level Classification: Identifies specific spans that are hallucinated
—RAG Optimized: Trained on diverse RAG benchmarks for broad applicability

Performance

Evaluated on RAGTruth test set (2,700 samples):

Metric	This Model	RAGTruth-Only	LettuceDetect BASE
Example-Level F1	77.00% ✅	77.49%	75.99%
Token-Level F1	53.37%	51.47%	56.27%
Training Data	48K samples	13.5K samples	13.5K samples
Context Window	32K	32K	8K

Key Results

—✅ Beats LettuceDetect BASE by +1% on example-level F1
—✅ 3.5x more training data (RAGTruth + HaluEval)
—✅ Better token-level F1 than RAGTruth-only model (53.37% vs 51.47%)
—✅ More diverse training - generalizes better to different task types

Related Model

—`modernbert-base-32k-haldetect` - Trained on RAGTruth only (77.49% F1)

Training Data

Dataset	Samples	Task Types
RAGTruth	~13,500	QA, Data-to-Text, Summarization
HaluEval	~38,700	QA, Summarization, Dialogue
Total	~48,400

HaluEval was converted from document-level to span-level annotations using NLI (DeBERTa-v3-mnli-fever-anli) and normalized to RAGTruth prompt format.

Usage

python

from transformers import AutoModelForTokenClassification, AutoTokenizer

model_name = "llm-semantic-router/modernbert-base-32k-haldetect-combined"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Format: context + question + answer
text = """Context: The Eiffel Tower is located in Paris, France.
Question: Where is the Eiffel Tower?
Answer: The Eiffel Tower is located in London, England."""

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=8192)
outputs = model(**inputs)
predictions = outputs.logits.argmax(dim=-1)

# 0 = Supported, 1 = Hallucinated

With LettuceDetect Library

python

from lettucedetect.models.inference import HallucinationDetector

detector = HallucinationDetector(
    method="transformer",
    model_path="llm-semantic-router/modernbert-base-32k-haldetect-combined"
)

context = "The Eiffel Tower is located in Paris, France."
question = "Where is the Eiffel Tower?"
answer = "The Eiffel Tower is located in London, England."

spans = detector.predict(context, question, answer)

Training Configuration

yaml

base_model: llm-semantic-router/modernbert-base-32k
datasets:
  - data/ragtruth/ragtruth_data.json
  - data/halueval_spans_normalized/halueval_data.json
max_length: 8192
batch_size: 8
learning_rate: 1e-5
epochs: 6
loss: CrossEntropyLoss
scheduler: None (constant LR)

Model Variants

Model	Training Data	Example F1	Best For
`modernbert-base-32k-haldetect`	RAGTruth only	77.49%	RAGTruth-style tasks
This model	RAGTruth + HaluEval	77.00%	General hallucination detection

Citation

bibtex

@misc{modernbert-32k-haldetect-combined,
  title={ModernBERT-base-32k Hallucination Detector (Combined)},
  author={llm-semantic-router},
  year={2025},
  url={https://huggingface.co/llm-semantic-router/modernbert-base-32k-haldetect-combined}
}

Acknowledgments

—Built on LettuceDetect framework
—Uses ModernBERT architecture
—Trained on RAGTruth and HaluEval datasets