ClarusC64/epistemic-confidence-calibration-v0.1

What this dataset does

This dataset tests whether a model can judge when high confidence is justified.

The task is simple:

Given a scenario and a confidence claim, predict whether the evidence supports high confidence.

Core stability idea

Reasoning fails when confidence rises faster than evidence quality.

This dataset targets that failure mode.

High confidence is justified when evidence is direct, repeated, independent, or clearly documented.

High confidence is not justified when evidence is weak, single-source, conflicted, uncontrolled, or causal without support.

Prediction target

Binary label:

• —1 = high confidence is justified
• —0 = high confidence is not justified

Row structure

Each row contains:

• —scenario_id
• —scenario_text
• —claim
• —label

Files

• —data/train.csv
• —data/test.csv
• —scorer.py
• —README.md

Evaluation

Create a predictions CSV with:

scenario_id,prediction
test_001,1
test_002,0

Run:

python scorer.py --predictions predictions.csv --truth data/test.csv

The scorer reports:

accuracy
precision
recall
f1
confusion matrix
Structural Note

This dataset is intentionally small.

Its purpose is to test whether a model can keep confidence proportional to evidence.

The hidden value is in detecting when repeated evidence, independent confirmation, conflict, sample size, and causal ambiguity change the justified confidence level.

License

MIT