latkes/self-consistency-correction-ambigqa

self-consistency-correction — AmbigQA

Run of the 5-score self-consistency-correction comparison on 10 hand-clustered AmbigQA questions from the rankalign longform branch data/ambigqa/with_negatives/. Paper: Correcting Generator Scores via Self-Consistency.

Setup

Each AmbigQA question has multiple valid answer entities (ambiguous interpretations) and rankalign-generated distractors (strategies: plausible, clearly-wrong, uninformed). We hand-clustered surface forms from the "Yes"-labeled rows into equivalence classes — one per interpretation entity — and kept 1-2 hand-selected distractors per question. All scoring uses the same teacher-forced log-prob machinery as the 20-case synthetic run.

Cases: plays, used, voice, received, harry, year, winter, american, soccer, executed. Total candidates: 171 across 10 questions (151 from 62 POS classes, 20 from 20 NEG singleton classes).

Pooled AUROC (score vs. iscorrectclass)

Pooled Spearman(score, V')

Interpretation

• —Negative-prompt (C) wins AUROC on both models. This matches the paper's §5.3 derivation: C factors into a discriminative likelihood ratio log P(a(y)|X, correct) − log P(a(y)|X, incorrect), whereas A/B/D are purely surface-form corrections. On a dataset where distractors are plausible (rankalign explicitly generates distractors by strategies like "plausible" and "clearly-wrong"), discrimination is what's needed, not surface-form defragmentation.
• —Raw G is anti-correlated with correctness on both models (AUROC 0.26 / 0.40). The model assigns high log-probability to plausible but wrong entities (e.g. "Ted Bundy" for the Utah firing squad question, "Sean Hannity" for "who used to be on The Five on Fox"). The generator-validator gap is real and directional.
• —Cluster correction is near-random on Qwen (0.42) and only mediocre on Llama (0.64). It inherits the raw-G bias on plausible distractors: if raw G puts high mass on "Sean Hannity" as a singleton class, log Σ exp(G) over that class is still just G(Hannity) — high — which wrongly ranks it above low-score singleton POS classes.
• —Cross-dataset comparison with the synthetic 20-case run: on synthetic cases where distractors were clearly wrong, cluster correction had AUROC 0.94-0.99. On AmbigQA with plausible distractors it drops to 0.42-0.64. The theoretical advantage of cluster correction is specifically for correcting surface-form fragmentation under a trusted validator — it does not help when the validator and generator are both fooled by the same plausible wrong meanings.
• —V' itself is only moderately accurate (AUROC 0.68 on both models). Specifically, on the executed case V' is 0.23 on Qwen (Ted Bundy and McVeigh trick it) and 0.55 on Llama. This is a separate failure mode of the framework: it assumes V' is a reliable meaning-level judge, which breaks when the model is confidently wrong about entity identity.

See notes/experiments/self-consistency-correction/FINAL_REPORT.md §3.6 for full writeup.