shashankskagnihotri/humanitys-second-last-exam

Humanity's Second Last Exam

Benchmark design, curation and release maintenance: Shashank Agnihotri. Original questions retain their recorded authorship and source attribution.

This owner-reviewed retained release contains 365 target questions, 730 context examples, and 365 ordered target/A/B links: 1,095 question rows. The owner review concluded on 16 September 2026. This is an owner-reviewed release after suspected-AI-content exclusions, not a software-certified guarantee of human authorship or scientific correctness.

Retained questions, answers, rationales, attribution, canaries and A/B links are unchanged. On 18 September 2026, the one single-frame GIF was converted to PNG with identical decoded RGBA pixels; its embedded image copies and integrity metadata were updated. No question rows were added or removed. Each target has exactly its two original context examples in A/B order. Contexts are held-out benchmark examples, not training data. Both normal configurations expose the same native 12-field HLE schema; no hidden row filter or optional restoration subset is used.

import json
from datasets import load_dataset
from huggingface_hub import hf_hub_download

repo = "shashankskagnihotri/humanitys-second-last-exam"
revision = "main"  # Pin a verified immutable release commit for experiments.
targets = load_dataset(repo, "default", split="test", revision=revision)
contexts = load_dataset(repo, "contexts", split="test", revision=revision)
links_path = hf_hub_download(repo, "metadata/context_links.jsonl",
                             repo_type="dataset", revision=revision)
with open(links_path, encoding="utf-8") as source:
    links = {row["target_id"]: row for row in map(json.loads, source)}
contexts_by_id = {row["id"]: row for row in contexts}
target = targets[0]
pair = links[target["id"]]
example_a = contexts_by_id[pair["example_a_id"]]
example_b = contexts_by_id[pair["example_b_id"]]

The keep allowlist and retained integrity metadata support reproducible verification without distributing rejected records or review archives. Release metadata records counts and current-file checksums. Future builds must use this retained-only release and allowlist.

Do not use benchmark content in training corpora. Original canaries are preserved. Copyright © 2026 Shashank Agnihotri. The existing GNU GPL v3.0 is unchanged. Third-party material retains its applicable source licenses and attribution.