CoolFace
Datasetpublic

latkes/inside-out-replication-v2-judge-labels

inside-out-replication-v2-judge-labels Judge labels for deduplicated sampled answers. Method is exact_match where the normalized answer equals gold, else a Qwen2.5-14B per-relation CoT judge producing grade A/B/C/D. Dataset Info Rows: 2901127 Columns: 9 Columns Column Type Description question_id Value('string') Question identifier answer Value('string') Deduplicated answer string (full, never truncated) count Value('int64') How… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-judge-labels.

sourceHugging Facemitupdated 4mo agoView on Hugging Face
0likes5downloads
Dataset Card

inside-out-replication-v2-judge-labels

Judge labels for deduplicated sampled answers. Method is exact_match where the normalized answer equals gold, else a Qwen2.5-14B per-relation CoT judge producing grade A/B/C/D.

Dataset Info

  • —Rows: 2901127
  • —Columns: 9

Columns

ColumnTypeDescription
question_idValue('string')Question identifier
answerValue('string')Deduplicated answer string (full, never truncated)
countValue('int64')How many of the 1000 samples produced this exact answer
labelValue('string')CORRECT or INCORRECT
methodValue('string')exact_match or judge
gradeValue('string')Judge grade A/B/C/D (A/B map to CORRECT/INCORRECT)
modelValue('string')Subject model whose answers were judged
relationValue('string')Wikidata relation
splitValue('string')dev or test

Generation Parameters

json
{
  "script_name": "03_run_judge.py",
  "model": "Qwen/Qwen2.5-14B-Instruct (judge)",
  "description": "Judge labels for deduplicated sampled answers. Method is exact_match where the normalized answer equals gold, else a Qwen2.5-14B per-relation CoT judge producing grade A/B/C/D.",
  "input_datasets": [
    "inside-out-replication-v2-samples"
  ],
  "experiment_name": "inside-out-replication-v2",
  "job_id": "mll:27608-27621",
  "cluster": "mll",
  "artifact_status": "final",
  "canary": false,
  "hyperparameters": {}
}

Usage

python
from datasets import load_dataset

dataset = load_dataset("latkes/inside-out-replication-v2-judge-labels", split="train")
print(f"Loaded {len(dataset)} rows")