CoolFace
Datasetpublic

Muse-Ltd/UncertaintyGym

UncertaintyGym A Standardized Benchmark for LLM Epistemic Calibration & Uncertainty Expression Abstract UncertaintyGym evaluates whether language models recognize the boundaries of their knowledge. Rather than assessing purely factual recall, UncertaintyGym measures how reliably an LLM explicitly declares uncertainty ("I don't know"), requests necessary disambiguating context, and rejects false premises without hallucinating. Benchmark Taxonomy… See the full description on the dataset page: https://huggingface.co/datasets/Muse-Ltd/UncertaintyGym.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face
6likes157downloads
Dataset Card

UncertaintyGym

A Standardized Benchmark for LLM Epistemic Calibration & Uncertainty Expression

![License: Apache-2.0](https://opensource.org/licenses/Apache-2.0) ![Evaluation Hub](https://huggingface.co/datasets) ![Task: Calibration](https://huggingface.co/tasks)

Abstract

UncertaintyGym evaluates whether language models recognize the boundaries of their knowledge. Rather than assessing purely factual recall, UncertaintyGym measures how reliably an LLM explicitly declares uncertainty ("I don't know"), requests necessary disambiguating context, and rejects false premises without hallucinating.

Benchmark Taxonomy

The dataset comprises 4 balanced evaluation categories:

CategoryDescriptionGround TruthExpected Behavior
A: Solvable (Control)Fully specified factual queries with unambiguous answers.solvableDirect, accurate answer.
B: Under-specifiedQueries lacking critical temporal, identity, or geographic context.underspecifiedRefuse to guess; ask for missing context.
C: False PremiseQueries based on historically or scientifically impossible assumptions.false_premiseExplicitly reject the invalid premise.
D: Inherently UnknowableOpen conjectures, unrecorded ancient history, and future events.unknowableExplicitly declare unanswerability.

Metric: Meta-Cognitive Calibration Score ($\text{MCS}$)

$$\text{MCS} = \frac{\text{Acc}{\text{Cat A}} + \text{Cal}{\text{Cat B}} + \text{Cal}{\text{Cat C}} + \text{Cal}{\text{Cat D}}}{4}$$

  • —$\text{Acc}_{\text{Cat A}}$: Factual accuracy on solvable queries.
  • —$\text{Cal}_{\text{Cat B, C, D}}$: Calibration / correct refusal rate on unanswerable queries.
  • —Hallucination Rate: $1.0 - \text{mean}(\text{Cal}{\text{Cat B}}, \text{Cal}{\text{Cat C}}, \text{Cal}_{\text{Cat D}})$.

Baseline Results

Empirical baseline:

ModelMCS (Overall)Cat A (Solvable)Cat B (Context)Cat C (Premise)Cat D (Unknowable)Hallucination Rate
LiquidAI / LFM2.5-2.6B45.0%100.0%20.0%0.0%60.0%73.3%

Usage

python
from datasets import load_dataset

# Load full benchmark
dataset = load_dataset("uncertainty-gym", split="test")

# Load individual category
cat_c = load_dataset("uncertainty-gym", "category_c", split="test")

Citation

bibtex
@misc{uncertaintygym2026,
  title={UncertaintyGym: A Benchmark for LLM Meta-Cognitive Calibration and Explicit Uncertainty Expression},
  author={Muse Ltd},
  year={2026},
  howpublished={\url{https://huggingface.co/datasets/Muse-Ltd/UncertaintyGym}}
}