Datasetpublic

Thorsu/sovereign-shadow-inference-bench

Sovereign Shadow Inference Bench A public, versioned evidence surface for independent Hugging Face shadow inference beside Sovereign's primary OpenRouter/Revolver route. What this dataset proves The seed record in data/shadow_receipts.jsonl was produced by one real Hugging Face Inference Providers request. It records provider/model identity, request bounds, latency, hashes, literal-match outcome, source revision, and an immutable receipt hash. What it… See the full description on the dataset page: https://huggingface.co/datasets/Thorsu/sovereign-shadow-inference-bench.

sourceHugging Faceapache-2.0updated 2d agoView on Hugging Face

1likes636downloads

Dataset Card

Sovereign Shadow Inference Bench

A public, versioned evidence surface for independent Hugging Face shadow inference beside Sovereign's primary OpenRouter/Revolver route.

What this dataset proves

The seed record in data/shadow_receipts.jsonl was produced by one real Hugging Face Inference Providers request. It records provider/model identity, request bounds, latency, hashes, literal-match outcome, source revision, and an immutable receipt hash.

What it does not prove

—Output agreement is not a semantic truth verdict.
—Hugging Face is not an automatic fallback and has no primary routing authority.
—A public record does not prove the current Sovereign VPS revision, Docker digest, PatchMon state, or deployment health.
—Missing provider cost metadata is reported as missing; it is never estimated into a verified claim.

Truth boundary

text

OpenRouter + Revolver  -> primary Sovereign inference authority
Hugging Face           -> independent shadow execution and public distribution
Sovereign runtime      -> authoritative deployment/runtime evidence
Public Space/Dataset   -> non-authoritative evidence projection

Public surfaces

—Space: Thorsu/sovereign-evidence-lab
—Dataset: Thorsu/sovereign-shadow-inference-bench
—Source repository: OuroborosCollective/Sovereign-Studio-ato

Read the public release note

Why “Agent says done” is not evidence

Current source-context canary

A second public canary was executed from an isolated worktree based on source revision 5dbe8a750b0d40f11cc3800ea091db945c9d2a49. It made exactly one nscale request, used no retry or fallback, matched the literal, and retained truthVerdict: NOT_ASSERTED. The receipt is `82f51e58b35cfe9bc9cf9bd734842a50fac25c7352851a8ea65ae9ace0093a67`.

This proves the bounded shadow request and receipt contract, not that the integration is merged or deployed in Sovereign.

Cross-provider and replication evidence — 2026-08-25 UTC / 2026-08-26 Europe/Berlin

The public corpus now contains 9 distinct actual-inference receipts across 2 model families and 4 providers. All nine receipts validate against the public schema and their canonical receipt hashes. Failed or partially conformant observations remain in the denominator.

Model	Provider	Budget	Runs	Literal matches	Latency observations	Evidence state
`Qwen/Qwen3-4B-Instruct-2507`	`nscale`	24	2	2/2	682, 705 ms	`REPEATED_ROUTE_CONFORMANCE_OBSERVED`
`Qwen/Qwen3-4B-Instruct-2507`	`featherless-ai`	24	2	2/2	5953, 4338 ms	`REPEATED_ROUTE_CONFORMANCE_OBSERVED`
`openai/gpt-oss-120b`	`cerebras`	64	2	2/2	332, 207 ms	`REPEATED_ROUTE_CONFORMANCE_OBSERVED`
`openai/gpt-oss-120b`	`groq`	64	2	1/2	250, 329 ms	`REPLICATION_MIXED_CONFORMANCE`
`openai/gpt-oss-120b`	`groq`	24	1	0/1	168 ms	`BUDGET_SENSITIVITY_OBSERVED`

Both model families remain CROSS_PROVIDER_OBSERVED, and the corpus remains MULTI_MODEL_OBSERVED with qualifier TWO_MODELS_CROSS_PROVIDER_OBSERVED. The replication round adds route-level evidence classified as ROUTE_REPLICATION_OBSERVED but does not create a model-quality, provider-speed, or truth leaderboard.

The new Groq 64-token replication used exactly one request with no retry or fallback. It spent 51 completion tokens on reasoning, reached finishReason: length, and emitted only a prefix of the expected literal. The previous Groq 64-token plan did emit the complete literal. This mixed result is preserved as route/budget envelope evidence rather than averaged into a fake green status.

The Cerebras replication completed the full literal even though the Hugging Face provider mapping preflight temporarily reported status=error; the real request therefore takes precedence as execution evidence for that observation. The mapping drift remains an availability signal, not a reason to rewrite the successful receipt.

For 2/2 route observations, the 95% Wilson interval for literal conformance remains roughly 34.2%–100%; for the Groq 64-token 1/2 observation it is roughly 9.5%–90.5%. These intervals show why the current corpus supports replication evidence, not stable reliability estimates.

Controlled Groq budget slice — 2026-08-26

A predeclared controlled slice held openai/gpt-oss-120b, provider groq, seed 424242, temperature 0, prompt and expected literal constant while changing only the output budget: 64 → 96 → 128. Each budget used exactly one request with zero retries and zero fallback.

Budget	Reasoning tokens	Literal	Finish	Latency
64	59	no visible literal	`length`	339 ms
96	59	exact match	`stop`	400 ms
128	59	exact match	`stop`	269 ms

Within this controlled slice, the first conformant budget observed was 96 tokens. Because the reasoning-token count stayed at 59 for all three requests and the 96/128 outputs are byte-identical, the observation supports an output-budget boundary between 64 and 96 for this exact slice. It does not establish a universal threshold, model-quality score, semantic-truth score, or provider-reliability ranking.

The existing shadow-inference-receipt.v1 schema has maxOutputTokens <= 64. Therefore only the 64-token observation is published as a v1 receipt. The 96/128 observations are preserved in the separately hash-bound controlled-slice record rather than being mislabeled as v1 receipts. See schema/controlled-budget-slice.v1.schema.json and data/slices/2026-08-26-groq-controlled-64-96-128-seed-424242.json.

Independent controlled Groq slice replication — seed 424243

A second predeclared 64/96/128 slice used the same model, provider, temperature and prompt template with independent fixed seed 424243. It did not reproduce the first slice's 96-token transition:

Seed	64	96	128	First conformant tested budget
424242	no	yes	yes	96
424243	no	no	yes	128

For seed 424243, reasoning usage was 62 tokens at budget 64 and 88 tokens at both 96 and 128. This changes the evidence state to CONTROLLED_BUDGET_BOUNDARY_SEED_VARIABILITY_OBSERVED: the existence of an output-budget boundary is reinforced, but a fixed universal threshold at 96 is not replicated. The two controlled slices support a seed/reasoning-dependent budget envelope, not a provider or model-quality ranking.

As before, receipt v1 permits at most 64 output tokens. The 64-token observation is a v1 receipt; 96/128 remain inside the separately hash-bound controlled slice.

Reproducible repository candidate

The repository integration candidate is published as a patch, Git bundle and changed-file archive under `artifacts/repository-candidate-37315b4448ac85b0bdc82ed8de59f358be9fdc09`. It is bound to base 5dbe8a750b0d40f11cc3800ea091db945c9d2a49 and candidate commit 37315b4448ac85b0bdc82ed8de59f358be9fdc09.

The candidate is locally verified but not claimed as a GitHub Draft PR, merge or Sovereign deployment while the private MCP tunnel is unavailable. Apply the patch only in a clone containing the exact base revision and rerun exact-head CI.

Record contract

The JSON Schema is stored at schema/shadow-inference-receipt.v1.schema.json. Public contributions should contain no API keys, private prompts, personal data, repository secrets, or unsupported runtime claims.

Research note: reasoning/output-budget failure mode

The two controlled Groq slices now have a concise public research note: `research-notes/2026-08-26-groq-reasoning-budget-boundary.md`. The key result is intentionally non-leaderboard: seed 424242 first conformed at 96 output tokens while seed 424243 first conformed at 128; both failed at 64 and both conformed at 128. This supports CONTROLLED_BUDGET_BOUNDARY_SEED_VARIABILITY_OBSERVED, not a universal threshold or provider/model ranking.

Requested multi-model matrix — 2026-08-25

A new bounded matrix records the requested Ox Alpha, Gemini 3.7 Flash, GPT-5.6 Sol/Terra, MiniMax M2.7, Claude Opus 5, DeepSeek, Llama, Claude Sonnet 4.5 and GLM 5.2 identities. Four official models were executable through HF Inference Providers; proprietary/stealth identities were recorded without substituting community distills. MiniMax M2.7 and GLM 5.2 both showed a 128-token length boundary and became literal-conformant at one separately recorded 256-token expansion point.

See `data/matrices/2026-08-25-requested-multimodel-canary.json` and `research-notes/2026-08-25-requested-multimodel-canary.md`. No intelligence/factuality leaderboard is inferred from this tiny heterogeneous matrix.

Publish-candidate provenance policy — 2026-09-18

The raw corpus remains immutable with 11 canonical-hash-verified receipts. The receipts viewer config is a flat projection of all 11 raw receipts so heterogeneous provider usage structures cannot break Arrow materialization. The publish_candidate config contains only the 2 receipts whose sourceRevision was independently resolved in OuroborosCollective/Sovereign-Studio-ato; the other 9 remain immutable historical evidence and are quarantined rather than relabeled. See data/provenance/source-revision-index.json.

The repository-candidate artifact identity is 37315b4448ac85b0bdc82ed8de59f358be9fdc09, proven by both the patch header and Git bundle head. This does not claim a GitHub merge or deployment.

Prepared model-expansion wave — 2026-09-18

A new publish candidate adds 11 plan-hash-bound actual-inference receipts across 9 model identities and 3 executed Hugging Face Inference Providers. The wave is anchored to Sovereign source revision df68cf8b7a16806cf19b4779abe469fdfdfc157a and dataset base revision d5740dcaaa700a5f54847f13855b5d34de65e30c.

The confirmatory cross-provider slice observed GLM-5.3 reaching finishReason=length without exact literal conformance on both Featherless and Novita at the fixed 64-token budget. DeepSeek-V4-Flash-0731 returned the exact public canary on both Featherless and DeepInfra. The same fixed plan also covers Kimi-K2-Instruct, Qwen3-Coder-30B-A3B-Instruct, Ornith-1.5-9B and a K2-Horizon size ladder (0.9B, 3.7B, 7B, MoVA-36B-A4B).

These are bounded execution observations, not a model-quality, intelligence, semantic-truth, speed or reliability leaderboard. The full plan, exact model revisions, provider mappings and receipts are under data/waves/2026-09-18/.

MiMo, Apertus and Grok-labelled community family waves — 2026-09-18

Three predeclared family plans extend the same evidence corpus. MiMo uses three official XiaomiMiMo repositories. Apertus uses four official Swiss AI repositories. The Grok-labelled wave uses four community repositories that contain Grok in their identity; it is explicitly not presented as official xAI Grok because no official xAI Grok repository exposed a live Hugging Face Inference Provider route during discovery.

Across the 11 planned routes, 10 produced actual inference and one MiMo-V2-Flash route did not complete. The failed route remains visible as a provider-route observation and is not converted into an actual-inference receipt. These observations do not form an intelligence, semantic-truth, quality, speed or reliability leaderboard.

Official frontier HF availability — 2026-09-18

Claude Fable 5.1, GPT-6 Astra and Gemini 3.8 Flash are recorded using their official vendor model identities. No official live Hugging Face Inference Provider route was found for any of the three at observation time, and the isolated run had no vendor API credentials. Therefore all three rows explicitly set actualInference=false. Community/distilled HF repositories were discovered where applicable but rejected as substitutes.

Meta, DeepSeek and Mistral confirmatory trio — 2026-09-18

A predeclared three-route confirmatory wave executed official Meta Llama 3.3 70B Instruct via Novita, DeepSeek V4 Pro 0813 via DeepInfra and Mistral Small 3.1 24B Instruct 2503 via Featherless. All three returned the exact public canary under the fixed 64-token plan. This is execution evidence, not a quality or intelligence ranking.

Daily benchmark wave — 2026-09-18

A predeclared four-task boundary wave ran once per task on one currently live official route each for Meta Llama, DeepSeek, Mistral AI, Apertus and MiMo. Route failures remain observations and are never converted into successful inference receipts. Task outcomes measure only the fixed machine-checkable boundary tasks; they are not intelligence, quality, or ranking claims.

Daily benchmark wave — 2026-09-19

Predeclared boundary tasks were executed once per route on official Meta Llama, DeepSeek, Mistral AI, Apertus and MiMo Hugging Face Inference Provider mappings. The wave records execution, route availability and machine-checkable task conformance only; it is not a quality, intelligence or ranking benchmark.

Daily benchmark wave — 2026-09-20

Predeclared boundary tasks were executed once per route on official Meta Llama, DeepSeek, Mistral AI, Apertus and MiMo Hugging Face Inference Provider mappings. Route failures remain observations and are never converted into successful inference receipts. The wave records execution, route availability and machine-checkable task conformance only; it is not a quality, intelligence or ranking benchmark.

Daily benchmark wave — 2026-09-21

Predeclared boundary tasks were executed once per route on official Meta Llama, DeepSeek, Mistral AI, Apertus and MiMo Hugging Face Inference Provider mappings selected before execution. Route failures remain observations and are never converted into successful inference receipts. The wave records execution, route availability and machine-checkable task conformance only; it is not a quality, intelligence or ranking benchmark.