nutrientdocs/grounding-en

1---2license: apache-2.03library_name: transformers4pipeline_tag: text-classification5language:6  - en7tags:8  - grounding9  - hallucination-detection10  - fact-verification11  - nli12  - zero-shot-classification13  - document-ai14  - cross-encoder15datasets:16  - nutrientdocs/grounding-benchmark17metrics:18  - roc_auc19---20 21# grounding-en22 23**Does the document actually support this claim?** `grounding-en` is a cross-encoder that scores whether24a hypothesis (a number, date, or fact) is **entailed by** a premise drawn from a real document — a25financial table, a filing, prose evidence.26 27It is the open, English member of Nutrient's grounding model family.28 29- 🎯 **Try it:** [grounding-demo](https://huggingface.co/spaces/nutrientdocs/grounding-demo?model=en)30- 🏆 **Leaderboard:** [grounding-leaderboard](https://huggingface.co/spaces/nutrientdocs/grounding-leaderboard)31- 📊 **Benchmark:** [grounding-benchmark](https://huggingface.co/datasets/nutrientdocs/grounding-benchmark)32 33## Results34 35On the held-out English [grounding-benchmark](https://huggingface.co/datasets/nutrientdocs/grounding-benchmark)36(ROC-AUC), against the strongest open English NLI models:37 38| Facet | `grounding-en` | `grounding-multilingual` | DeBERTa-v3-large zero-shot | DeBERTa-v3-large MNLI/FEVER/ANLI | BART-large MNLI |39| --- | ---: | ---: | ---: | ---: | ---: |40| **Overall** | **.882** | .925 | .786 | .769 | .636 |41| Number | **.923** | .969 | .658 | .642 | .478 |42| Date | **.998** | .999 | .995 | .995 | .924 |43| String | **.955** | .949 | .941 | .913 | .757 |44| Table premises | **.863** | .915 | .766 | .747 | .611 |45| Prose premises | **.964** | .970 | .929 | .945 | .892 |46 47`grounding-en` leads the field on the hard axis — **number grounding .92** vs .48–.66 for general-purpose48NLI models — while matching or beating them everywhere else. Full ranking on the49[leaderboard](https://huggingface.co/spaces/nutrientdocs/grounding-leaderboard). The commercial sibling50[`grounding-multilingual`](https://huggingface.co/nutrientdocs/grounding-multilingual) scores a touch51higher again and covers 15+ languages.52 53## Usage54 55```python56import torch57from transformers import AutoModelForSequenceClassification, AutoTokenizer58 59m = "nutrientdocs/grounding-en"60tok = AutoTokenizer.from_pretrained(m)61model = AutoModelForSequenceClassification.from_pretrained(m).eval()62 63premise = "Revenue | 2023 | $4,213M\nRevenue | 2022 | $3,905M"64hypothesis = "2023 revenue was $4.2 billion."65 66enc = tok(premise, hypothesis, truncation=True, max_length=1024, return_tensors="pt")67with torch.no_grad():68    probs = torch.softmax(model(**enc).logits, dim=-1)[0]69p_support = probs[0].item()   # entailment is class index 0 (id2label = {0: entailment, 1: not_entailment})70print(f"grounded support = {p_support:.3f}")71```72 73An **ONNX** export is provided under [`onnx/`](./onnx) for on-device / ONNX Runtime deployment.74 75## Calibrating the score76 77Fine-tuning maximizes _ranking_ (AUC), which tends to make the raw probability overconfident. For a78score you can gate on ("0.9 means ~90% right"), apply **temperature scaling** — divide the logits by a79fitted `T` before softmax. It's monotonic, so it leaves AUC/ranking untouched and only repairs the80confidence values. On the serving distribution we fit **T = 1.29** (ECE 0.028 → 0.009). Re-fit `T` on81_your_ distribution whenever your input pipeline changes.82 83## Intended use & limits84 85- **Use it for:** verifying extracted values against source documents, hallucination/citation checking,86  routing low-confidence extractions for review.87- **Limits:** English only. The remaining ceiling is _reasoning_ table-claim negatives and88  multi-step arithmetic. As a dedicated grounding model it trades a little general-NLI accuracy for89  grounding.90 91## License & training data92 93Weights are **Apache-2.0**. Trained on a multi-corpus grounding set. The public,94redistributable slice of the _evaluation_ data is95[grounding-benchmark](https://huggingface.co/datasets/nutrientdocs/grounding-benchmark) (CC-BY-SA-4.0);96the full training set is not redistributed.97 98## About the author99 100<a href="https://nutrient.io/">101  <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" />102</a>103 104This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.105