Thorsu/sovereign-evidence-observatory

Sovereign Evidence Observatory Casebook

The interesting question is not “Which AI sounds smartest?” It is “What kind of evidence would make this claim true, false, or still undecidable?”

This public casebook is the first dataset for the Sovereign Evidence Observatory. It turns model agreement, disagreement and abstention into inspectable evidence objects instead of treating consensus as truth.

Five layers

1. 1.Shadow Mesh — independent model/provider observations.
2. 2.Disagreement Radar — records where outputs agree, diverge or answer different questions.
3. 3.Proof Router — routes each claim to the evidence class that can actually verify it: formal computation, structured data, source provenance, target-system runtime, or human judgment.
4. 4.Abstention Gate — missing evidence becomes UNPROVEN, not a fake green status.
5. 5.Evidence Passport — a portable, hash-bound summary of claim, observations, verifier class, evidence and verdict.

Why this is useful

A multi-model vote can improve robustness, but agreement alone cannot establish truth. Research on multi-agent debate reports factuality gains from independent agents, while work on semantic entropy shows that uncertainty can help detect confabulations. Recent fact-verification work also highlights the importance of executable evidence and the difficulty models have with abstention.

The Observatory therefore separates model consensus from evidence class and verification state. A result can be PROVEN, CONTRADICTED, UNPROVEN, or NOT_APPLICABLE without pretending that every claim has a machine-verifiable truth condition.

Public companion

Interactive Space: Thorsu/sovereign-evidence-observatory

Existing live shadow receipts: Thorsu/sovereign-shadow-inference-bench

Current Hugging Face participant evidence

The public Shadow Bench now represents 15 distinct actual inference requests: 11 are compatible with the legacy shadow-inference-receipt.v1 contract and four additional 96/128-token requests are preserved in two separately hash-bound controlled budget slices.

Two independent Groq slices used openai/gpt-oss-120b, temperature 0, the same prompt template and the same 64/96/128 ladder with fixed seeds 424242 and 424243. Both failed to emit the bounded literal at 64 and both emitted it at 128, but the first conformant tested budget differed: 96 for seed 424242 versus 128 for seed 424243. The current evidence state is therefore CONTROLLED_BUDGET_BOUNDARY_SEED_VARIABILITY_OBSERVED, not a universal 96-token threshold.

Qwen and gpt-oss remain CROSS_PROVIDER_OBSERVED; route-replication evidence remains public, including all non-conformant observations. No participant leaderboard or provider-quality ranking is enabled from this small heterogeneous corpus.

See metrics/HF_PARTICIPANT_EVALUATION_2026-08-26.json for the current machine-readable snapshot bound to Shadow Bench revision 9ee746234cf1292b639744298af551959ada2f35.

Public failure-mode story

The interactive Space now has a Budget Boundary Lab that re-reads and hash-verifies the two public Groq 64/96/128 slices without performing new inference. The public research note is in the Shadow Bench at research-notes/2026-08-26-groq-reasoning-budget-boundary.md.

The machine-readable participant evaluation remains historically bound to Shadow Bench revision 9ee746234cf1292b639744298af551959ada2f35, where the evaluated slice data was published. The later Shadow Bench revision c93877086acc70a64b27a42b036c88456141436a is a documentation-only research-note commit; it does not rewrite the underlying receipts or controlled slices.

Public entry points:

• —Space: https://huggingface.co/spaces/Thorsu/sovereign-evidence-observatory
• —Shadow Bench: https://huggingface.co/datasets/Thorsu/sovereign-shadow-inference-bench
• —Research note: https://huggingface.co/datasets/Thorsu/sovereign-shadow-inference-bench/blob/main/research-notes/2026-08-26-groq-reasoning-budget-boundary.md

Research anchors

• —Du et al., ICML 2024 — Improving Factuality and Reasoning in Language Models through Multiagent Debate.
• —Farquhar et al., Nature 2024 — Detecting hallucinations in large language models using semantic entropy.
• —ClaimDB, ACL 2026 — large structured-data fact verification and abstention.
• —UniFact, 2025 — complementary roles of hallucination detection and fact verification.

Truth boundary

This dataset contains both real receipts and explicitly labeled synthetic/conceptual demonstrations. Synthetic model answers are never presented as actual provider outputs. Public receipts prove only the execution or evidence event they record.

Proof Router V2

The public Space now includes executable evidence routing across formal computation, structured data, source provenance, runtime readback, human judgment and abstention. See `PROOF_ROUTER_V2.md`, the V2 schemas under schema/, and reproducible example receipts under examples/.

V2.0.1 stability evidence

The exact-revision live stabilization readback is published at data/proof-router-v2-stability-live-readback.json. It separates unit tests, live API soak/burst results, Hub runtime revision/state, and remaining truth boundaries.

Requested multi-model canary matrix

The public Shadow Bench now contains a bounded ten-name identity/execution matrix covering Ox Alpha, Gemini 3.7 Flash, GPT-5.6 Sol/Terra, MiniMax M2.7, Claude Opus 5, DeepSeek, Llama, Claude Sonnet 4.5 and GLM 5.2. Four official models were executable through Hugging Face Inference Providers. Proprietary/stealth identities were recorded without substituting similarly named community distills.

At the primary 128-token budget, DeepSeek V4 Flash 0731 and Llama 4 Maverick produced the exact literal; MiniMax M2.7 and GLM 5.2 ended at length with no visible literal, then both conformed at the single separately recorded 256-token expansion. This is budget/route evidence, not a model leaderboard. See metrics/HF_MULTIMODEL_CANARY_MATRIX_2026-08-25.json.