kernel-14/SemanticAlign-Bench

SemanticAlign-Bench

A benchmark for evaluating AI agents on structured claim extraction from top-tier ML conference papers. Each paper is decomposed into Semantic Alignment Units (SAU) — atomic, self-contained implementation propositions — across four diagnostic dimensions spanning numerical precision to pipeline-level workflow. Agents are evaluated on whether they can reproduce these claims without hallucination, omission, or misordering.

Dataset Description

• —Papers: 30 papers from ICLR 2025, ICML 2025, and NeurIPS 2025, spanning 5 domains (6 papers each):

• —Total SAU Claims: 1,491
• —Size: ~519 MB

The Four SAU Dimensions

Each paper is decomposed into claims across four diagnostic dimensions, ordered from micro to macro:

The D1--D4 hierarchy is universal across all evaluated configurations: D1 > D2 > D4 > D3 in score holds invariant for all 12 generator setups (Claude/DeepSeek/Gemini/GPT-4o × BasicAgent/PaperCoder/OpenHands). D3 (experimental protocol) is the dominant bottleneck, with only 0.7% perfect-score rate — 14× lower than D1. D4 exhibits a distinctive pattern: lowest zero rate (33.7%) but only 5.9% of claims score ≥0.5, meaning agents almost always attempt ordering constraints but rarely get them right.

Paper Venue Distribution

Dataset Structure

Per-Paper Directory Layout

<paper_id>/
  config.yaml       # Paper metadata (title, venue, year, domain, arxiv URL)
  paper.md          # Full paper text in markdown
  paper.pdf         # Original PDF
  sau.json          # SAU claims — the core annotation file
  images/           # Paper figures extracted from PDF
  blacklist.txt     # official repo url

SAU Claim Format (sau.json)

{
  "paper_id": "adjoint-matching",
  "paper_title": "Adjoint Matching: Fine-tuning Flow and Diffusion Models with Memoryless SOC",
  "D1": [
    {
      "id": "adjoint-matching-D1-001",
      "claim": "Image resolution for autoencoder pre-training and generation: 512×512",
      "source": "Section 7"
    }
  ],
  "D2": [ ... ],
  "D3": [ ... ],
  "D4": [ ... ]
}

Each claim includes:

• —id: Unique identifier ({paper}-{dimension}-{number})
• —claim: Self-contained implementation proposition in natural language
• —source: Paper section where the claim originates

Annotation Quality

All 1,491 claims have undergone multi-version human review with systematic error checks:

• —Verification against source paper for factual accuracy
• —Format normalization and consistency validation
• —Cross-reference integrity checks between dimensions
• —Fairness audit across domains and paper types (theory vs. empirical)

Supported Tasks

1. 1.Claim-Level Factuality: Given a paper, can the agent accurately extract a specific numerical value, formula, experimental detail, or procedural step?
2. 2.Dimension-Level Completeness: Can the agent achieve full recall across all four SAU dimensions for a given paper?
3. 3.Cross-Dimensional Consistency: Are claims in D4 (pipelines) consistent with D2 (formulas) and D3 (experiments)?
4. 4.Hallucination Detection: Can the agent distinguish paper-supported claims from plausible but fabricated ones?

Dataset Creation

Source Data

30 papers selected from ICLR 2025, ICML 2025, and NeurIPS 2025, covering 5 domains with equal representation across task types (classification, generation, RL, theory, scientific computing).

Evaluation Results

In a benchmark study evaluating 360 paper-level runs (12 generators × 30 papers):

• —Overall SAS: mean 0.221, median 0.200. 82.4% of SAU claims score ≤0.25.
• —Model dominance: Model choice drives 2.35× more score variation than scaffold choice (1.15×). Top 5 configurations all use Claude or DeepSeek; bottom 3 all use GPT-4o.
• —Scaffold asymmetry: PaperCoder (+0.116 for GPT-4o) provides more benefit to weaker models. OpenHands adds near-zero value without minimum planning competence.
• —Failure pattern: 81% of zero-scored claims contain partial but incorrect code; only 5.7% are completely absent. Improving scores requires better comprehension, not broader coverage.
• —Paper difficulty: Numerical methods/PDE papers dominate the easiest tier; multi-modal systems and complex training pipelines the hardest.

Considerations for Using the Data

Limitations

This is a static benchmark: claims test specification fidelity (did the agent encode the right parameters, formulas, and protocols?) rather than runtime correctness. The benchmark does not include execution-based evaluation or dynamic testing.

Intended Use

• —Benchmarking LLM factuality on scientific content
• —Measuring agent understanding of structured paper content
• —Stress-testing retrieval-augmented generation (RAG) over academic papers

Out-of-Scope Uses

• —Training data for production LLMs (limited size, single annotator)
• —Automated paper review or acceptance prediction

Additional Information

License

SAU annotations are licensed under CC-BY-4.0. Underlying papers are subject to their original copyright terms as posted on arXiv and respective conference proceedings.

Citation

@inproceedings{semanticalign_bench,
  title     = {SemanticAlign-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction},
  author    = {Anonymous Author(s)},
  year      = {2025},
  note      = {Benchmark dataset at \url{https://anonymous-hf.up.railway.app/a/rrgn430zpfui/}}
}

Papers List