NagaYu/claimcheck-rules

ClaimCheck Rule Pack v1.1.0

### ⚠️ This is not a neural model. There are no weights in this repository. No training was performed, no gradients were computed, and nothing here is loadable with transformers despite what the sidebar tag implies — the Hub requires a library_name and there is no tag for "regexes and arithmetic". What this repository actually holds is a versioned rule pack: the regular expressions, normalisation tables, suppression lists and policy presets that drive ClaimCheck, a deterministic LLM answer verifier. It is published here so the rules can be inspected, diffed, cited and versioned independently of the app. Presenting it as a trained model would be a false claim, so it isn't.

Related: Space · evaluation set

What the rules do

ClaimCheck verifies an LLM answer against the context that was supplied to it. It extracts claims it can check as strings — numbers, dates, quotes, entities, URLs — normalises both sides, and assigns each claim one status.

The design premise, in one line:

*We cannot detect every hallucination. But dangerous hallucinations are specific, and specific claims can be matched as strings.*

So it verifies only what it can verify deterministically, and always reports how much it did not check (coverage).

Everything runs on pure Python with zero dependencies for verification (pandas and gradio are for the dashboard and UI only). Typical verification is 0.17 ms; a 20,000-character worst case is 99 ms on 2 vCPU.

Contents

rules/
  manifest.json         versions, claim types, statuses, verdicts, engine facts
  numeric.json          numeric recognition, magnitude factors (千/万/億/兆, k/M/B)
  dates.json            date patterns, Japanese era offsets, month names
  entities.json         entity patterns, 200+ stopwords, ISO currency codes
  quotes.json           quote patterns and similarity thresholds
  output_safety.json    credential prefixes, Luhn, JWT, entropy thresholds
  leak_detection.json   system-prompt leak and prompt-injection echo patterns
  sentences.json        sentence segmentation — the coverage denominator
policies/
  01-observe.json               start here: nothing blocked but critical safety
  02-numeric-date-only.json     lowest false-positive configuration
  03-retry-on-contradiction.json first setting that costs a retry
  04-strict.json                full enforcement — do NOT start here
  05-low-latency.json           for constrained CPU / high throughput
  06-tolerant-numbers.json      for sources that round

Every file in rules/ is generated programmatically from `python/app.py`, so it cannot drift from the code it documents.

The hosted Space runs a JavaScript port of the same rules, verified at 100% behavioural parity with the Python implementation across all 158 evaluation cases. Use python/ when you want a server-side REST API; use the Space when you want a zero-install browser page that sends nothing anywhere.

Usage

The policy presets are directly usable — they are exactly the policy_json argument the Space's /verify endpoint takes.

from huggingface_hub import hf_hub_download
from gradio_client import Client
import json

policy = json.load(open(hf_hub_download(
    "NagaYu/claimcheck-rules", "policies/02-numeric-date-only.json")))
policy = {k: v for k, v in policy.items() if not k.startswith("_")}  # drop docs

client = Client("NagaYu/ClaimCheck")
result, highlighted, summary = client.predict(
    answer="Operating margin was 15%.",
    context="Revenue was 12,000 million and operating profit 1,800 million.",
    system_prompt="", user_input="", schema_json="",
    policy_json=json.dumps(policy),
    tags_json='{"model": "my-model", "prompt_version": "v1"}',
    api_name="/verify",
)
print(result["verdict"], result["grounding_score"], result["coverage"])

Inspecting a rule:

rules = json.load(open(hf_hub_download("NagaYu/claimcheck-rules", "rules/numeric.json")))
print(rules["magnitude_factors"])   # {'千': '1E+3', '万': '1E+4', '億': '1E+8', ...}

Staged rollout

The presets are numbered because the order matters. A verifier loses its users the first time it cries wolf, and an unaudited one at full strength will.

Do not start at stage 4.

Evaluation

Measured on `NagaYu/claimcheck-eval` (158 bilingual cases):

spec conformance         158/158  (100%)
answer-level recall       56/56   (100%)   cases that should be flagged, and were
clean answers passing     44/46   (96%)
known false positives     22      (13.9%)
known false negatives     3       (1.9%)
latency                   p50 0.17ms · p95 0.39ms · max 0.96ms

Conformance is 100% by construction and is not a quality score — the labels were reconciled against these rules. The informative numbers are the false positive and false negative rates.

Known failure modes

Japanese semantic reversal is the most serious. A one-character change that reverses meaning keeps character similarity above 0.9:

Character matching cannot detect negation or antonym substitution. If your domain turns on approve/reject or increase/decrease, do not rely on quote status alone — the NUMERIC and ENTITY claims inside a quote are the layer that catches these in practice (all three still produced verdict: retry).

False positives (13.9%) cluster in faithful paraphrase, generic noun phrases ("Machine Learning"), acronym expansion (WHO ↔ World Health Organization), rounding just outside tolerance, fiscal vs calendar years, and URL query parameters. ENTITY is the noisiest check — disable it first.

Bugs this rule pack fixed in v1.1.0

Found by running the evaluation set, not by unit tests:

1. 1.\b never fires between a CJK character and a digit → ISO dates inside Japanese text were invisible.
2. 2.U+30FC (katakana prolonged sound mark) was folded to a hyphen → データセンター became デ-タセンタ-, breaking Japanese matching wholesale.
3. 3.An exact sign flip was excused by a coincidental derivation → a loss reported as a profit returned verdict: pass.
4. 4.Leading whitespace absorbed into numeric matches, breaking list-marker suppression and span accuracy.
5. 5.ISO currency codes claimed as entities → `USD 45,000` lost its number.
6. 6.Bare years in the answer never compared against context dates.
7. 7.Parenthesised accounting negatives (1,200) read as positive.

Four of the seven affect Japanese or financial text specifically.

Intended use

Use it for: flagging specific, checkable claims in RAG and summarisation output; catching fabricated citations, figures and dates before they reach users; measuring groundedness across prompt versions; building a targeted retry instead of a blind one.

Do not use it for: general fact-checking against world knowledge (it only knows the context you hand it), judging reasoning, tone or completeness, or as a sole gate on high-stakes output. It says nothing about claims it could not extract — that is what coverage is for.

Bias and risks

• —Asymmetric language coverage. Japanese support is deliberate and tested, but character-level similarity behaves differently across scripts; the semantic-reversal failure is more severe in Japanese, where negation is often a single character.
• —A `pass` verdict is not a correctness guarantee. It means the specific things this tool knows how to check, checked out.
• —Low `coverage` with a high `grounding_score` is the dangerous reading. An answer of pure hedging produces zero claims and scores 1.00 on any naive groundedness metric while being completely unverified.
• —Verification is local and deterministic; nothing is sent anywhere. The one optional remote call (a relevance score) is off without a token and never affects the verdict.

Citation

@misc{claimcheck_rules_2026,
  title  = {ClaimCheck Rule Pack: deterministic claim-level verification for LLM answers},
  author = {NagaYu},
  year   = {2026},
  url    = {https://huggingface.co/NagaYu/claimcheck-rules}
}

Apache-2.0.