CoolFace
Datasetpublic

latkes/inside-out-replication-v2-external-scores

inside-out-replication-v2-external-scores Per (question, answer) external scores with judge labels for Inside-Out V2. Covers P(a|q), P_norm(a|q), P(True) and two verification-prompt variants. Used to compute external K/K*. Dataset Info Rows: 1752198 Columns: 20 Columns Column Type Description question_id Value('string') Question identifier, e.g. P26_test_0000 answer Value('string') Full sampled answer text (never truncated) label… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-external-scores.

sourceHugging Facemitupdated 4mo agoView on Hugging Face
0likes4downloads
Dataset Card

inside-out-replication-v2-external-scores

Per (question, answer) external scores with judge labels for Inside-Out V2. Covers P(a|q), P_norm(a|q), P(True) and two verification-prompt variants. Used to compute external K/K*.

Dataset Info

  • —Rows: 1752198
  • —Columns: 20

Columns

ColumnTypeDescription
question_idValue('string')Question identifier, e.g. P26test0000
answerValue('string')Full sampled answer text (never truncated)
labelValue('string')Judge label: CORRECT or INCORRECT
logpa_qValue('float64')Log prob of answer given question
paqValue('float64')exp(logpa_q)
logpnormaqValue('float64')Length-normalized log prob
pnorma_qValue('float64')exp(logpnormaq)
p_trueValue('float64')Restricted softmax P(A=CORRECT) over {A,B} verification tokens
ptruefull_aValue('float64')Full-vocab prob mass on the 'A' token
ptruefull_bValue('float64')Full-vocab prob mass on the 'B' token
ptrueresidualValue('float64')1 - (full A + full B): mass outside the A/B tokens
verifv0ab_scoreValue('float64')P(True) variant: A/B prompt (same as p_true)
verifv0ab_residualValue('float64')Residual mass for v0 A/B prompt
verifv1truefalse_scoreValue('float64')Verification variant: True/False prompt
verifv1truefalse_residualValue('float64')Residual mass for True/False prompt
verifv2yesno_scoreValue('float64')Verification variant: Yes/No prompt
verifv2yesno_residualValue('float64')Residual mass for Yes/No prompt
modelValue('string')Subject model
relationValue('string')Wikidata relation (P26/P264/P176/P50)
splitValue('string')dev or test

Generation Parameters

json
{
  "script_name": "04_external_scores.py",
  "model": "Llama-3-8B / Mistral-7B-v0.3 / Gemma-2-9B",
  "description": "Per (question, answer) external scores with judge labels for Inside-Out V2. Covers P(a|q), P_norm(a|q), P(True) and two verification-prompt variants. Used to compute external K/K*.",
  "input_datasets": [
    "inside-out-replication-v2-judge-labels"
  ],
  "experiment_name": "inside-out-replication-v2",
  "job_id": "mll:27608-27621",
  "cluster": "mll",
  "artifact_status": "final",
  "canary": false,
  "hyperparameters": {}
}

Usage

python
from datasets import load_dataset

dataset = load_dataset("latkes/inside-out-replication-v2-external-scores", split="train")
print(f"Loaded {len(dataset)} rows")