Datasetpublic

karantonis/llada2-mini-uq-pickles

LLaDA 2.1 mini — UQ eval pickles (ue_manager_seed1) Durable backup of the LLaDA 2.1 mini uncertainty-quantification sweep pickles produced with lm-polygraph (branch feat/llada2-cache, April–May 2026). Each .pkl is a torch.load-able ue_manager dump containing per-sample stats (greedy_texts, target_texts, ...), estimations, gen_metrics, and metrics. Layout setup_a/ no-train baselines, reduced config (~34 est, K=10 sampling dropped) setup_b/ with-train baselines… See the full description on the dataset page: https://huggingface.co/datasets/karantonis/llada2-mini-uq-pickles.

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes63downloads

Dataset Card

LLaDA 2.1 mini — UQ eval pickles (`ue_manager_seed1`)

Durable backup of the LLaDA 2.1 mini uncertainty-quantification sweep pickles produced with lm-polygraph (branch feat/llada2-cache, April–May 2026). Each .pkl is a torch.load-able ue_manager dump containing per-sample stats (greedytexts, targettexts, ...), estimations, gen_metrics, and metrics.

Layout

setup_a/   no-train baselines, reduced config (~34 est, K=10 sampling dropped)
setup_b/   with-train baselines (79 est)
setup_c/   train-only estimators (6 est: TAD/MIND/SAPLMA/SATRMD/LookBackLens/Mahalanobis)
setup_d/   diffusion methods, merged config (542 est incl. ConfidenceAtCommit family)

Coverage (8 datasets in scope + truthfulqa)

Dataset	setup_a	setup_b	setup_c	setup_d
coqa	—	—	—	✅
gsm8k	—	—	—	✅
mmlu	✅	—	✅	✅
samsum	✅¹	—	—	✅ (strip-fixed)
triviaqa	✅	✅	—	✅
wmt14_fren	✅	—	—	✅
wmt19_deen	✅	—	—	✅
xsum	✅¹	—	—	✅ (strip-fixed)
truthfulqa	—	—	—	✅ (descoped)

Caveats

—Metrics: MCQ/QA (mmlu, coqa, triviaqa) strict Accuracy undercounts (verbose answers / "B. …" prefixes). Recompute with scripts/repatch_pickle_accuracy.py for real numbers (e.g. mmlu 4.8% → 33%). NMT (wmt14/wmt19) — use Comet (~0.80), not Accuracy.
—¹ samsum/xsum setup_a carry a high meta-prefix rate (~44–66% of outputs begin with "It seems…"), but a real summary is usually embedded after the prefix, so AlignScore (~0.66) is the usable signal.
—Missing: coqa & gsm8k setupa; setupb beyond triviaqa (all other setupb runs were failed stubs). setupc is derivable from setup_b/d hidden states where present.

Provenance

Generated on MBZUAI cluster (mbz1–4) + CIAI, model inclusionAI/LLaDA2.1-mini, DAE decoding (tau_edit=0.0, mbe_iters=16, suppress_eos_min_total_committed=50), subsample_eval_dataset=2000. See PR ArtemVazh/uncertainty_dllm#4.