agentGreg/czy-bielik-zna-temat

Czy Bielik zna temat? / Does Bielik know the topic?

Zadaj polskie pytanie o encję ("Kim jest X?" / "Czym jest X?"). Model Bielik-11B-v3.0-Instruct odpowie, a liniowy probe czytający jego wewnętrzne aktywacje przy ostatnim tokenie pytania oszacuje skalibrowane ryzyko, że model tematu nie zna i konfabuluje.

Ask a Polish entity question; a linear probe over the model's last-prompt-token activations returns a calibrated P(hallucination-from-ignorance). The probe never sees the generated answer, so the risk estimate is invariant to decoding settings.

Hardware

• —Required: zerogpu. Hosting a ZeroGPU Space needs an HF PRO account (personal) or a Team/Enterprise plan (org). The large slice (48 GB) fits Bielik-11B in bf16 (~22 GB) with room to spare, so the default @spaces.GPU size is correct — do not request xlarge (2× quota cost).
• —Quota: ZeroGPU GPU-seconds are billed to the visitor, not the Space owner (unauthenticated 2 min/day, free account 5 min, PRO 40 min). Traffic spikes therefore do not bill the owner.
• —Why 11B and not 1.5B: the probe separates known/unknown at ~0.99 AUROC at every size, but answer correctness scales with parameters — full correctness (5/5 attempts) is 19/42 at 11B vs 0/42 at 1.5B. On 1.5B the demo would routinely show "LOW risk" next to a wrong answer, which misrepresents what the probe claims.

ZeroGPU contract (do not break)

app.py places the model on cuda at module level and decorates only the forward+generate step with @spaces.GPU(duration=60). This is deliberate:

• —Outside @spaces.GPU, CUDA is emulated, and HF optimizes CUDA transfers for startup placement.
• —Lazy-loading inside the decorated function would bill the ~22 GB cold start to the first visitor's GPU quota and likely blow the 60 s duration cap.

Secrets

The Bielik v3.0 models are gated. Before the Space can download the model:

1. 1.Accept the model license at https://huggingface.co/speakleash/Bielik-11B-v3.0-Instruct
2. 2.In Space Settings → Variables and secrets, add a secret named `HF_TOKEN` whose value is a Hugging Face token with access to that model. Prefer a fine-grained read token scoped to that one model repo over a full-access token.

If HF_TOKEN is missing or lacks access, the app surfaces a clear error in the UI instead of crashing (the startup load captures failures rather than raising).

What to copy into the Space repo

app.py inserts src/ on sys.path and resolves the probe path relative to itself, so the layout above is all that is needed — no packaging/install step. The src/bielik_hallu/risk/ folder must contain __init__.py, probe.py, and inference.py.

Deployment is scripted — from the repo root:

uv run python scripts/deploy_space.py            # private by default
uv run python scripts/deploy_space.py --public   # flip to public when happy

Examples

The bundled examples are all outside the probe's 504-entity training pool (verified against data/Bielik-11B-v3.0-Instruct/**/labeled.parquet), so the demo is not scoring its own training data. They walk through the conditions the paper separates, with scores measured on this Space:

Płatkowieński is the demo that lands: the probe says 100% risk and the model still answers that he was mayor of Olsztyn. Note Czerwionka-Leszczyny was expected to be UNKNOWN_REAL and is not — 11B genuinely knows it, so the label was wrong, not the probe. The haiku prompt is off-template and included to show where calibration stops.

Honest disclaimer

This measures the model's familiarity with the entity/topic, not factual correctness. A LOW score means "the model is familiar", not "the answer is correct in every detail" — recognition is not reliability. It is calibrated on one-sentence Polish entity questions and is experimental off-template (reasoning, creative writing, non-entity questions). It is a risk gate, not a fact-checker. See the "How it works" panel in the app.