KRLabsOrg/acl-verbatim-spans

ACL-Verbatim Span Dataset

`KRLabsOrg/acl-verbatim-spans` is a dataset for query-conditioned extractive evidence selection over papers from the ACL Anthology.

The release combines:

• —a gold test benchmark with manual span annotations
• —a larger silver training set produced from synthetic questions, retrieval, and LLM-based span annotation
• —an encoder-ready config for training token-classification models directly

The underlying document collection is `KRLabsOrg/acl-anthology-md`.

What This Dataset Is For

This dataset is intended for systems that, given a question and a retrieved paper chunk, must identify the supporting evidence verbatim in the chunk.

Typical uses include:

• —training token classifiers for semantic highlighting
• —evaluating span extractors and evidence selectors
• —comparing LLM teachers, token-level students, and sentence-selection baselines
• —studying paragraph-scale evidence extraction in scientific text

Configs

canonical

One row per (question, chunk) pair.

• —train and validation are silver supervision
• —test is the manual gold benchmark

This is the main config for analysis, evaluation, and downstream reuse.

Current split sizes:

Expected fields:

encoder

Token-classification-ready rows derived from the silver canonical data by tokenization and windowing.

This config is intended for direct encoder training with Hugging Face transformers.

Current split sizes:

Expected fields:

How the encoder split was built

The encoder split was pretokenized with:

If you want to train with a different tokenizer or label scheme, rebuild from the canonical config:

python acl_verbatim/span_training/prepare_token_cls_dataset.py \
  --input-file <canonical_train.jsonl> \
  --output-file train.my_tokenizer.binary.jsonl \
  --tokenizer <your-tokenizer> \
  --label-scheme binary \
  --drop-unlabeled-positives

Evaluation always uses canonical/test (raw text); the encoder config intentionally does not include a test split, because span-level scoring needs to map token predictions back to character offsets in the original chunk, which requires retokenizing at inference time.

Annotation Convention

The benchmark uses paragraph-oriented evidence annotation rather than minimal SQuAD-style answer spans.

Important consequences:

• —broader supporting passages are often preferred over minimal snippets
• —tables, figure captions, and other structured evidence are considered valid positives
• —bibliography/reference sections are generally out of scope as positive evidence

Gold Benchmark Summary

The canonical/test split is the manually annotated benchmark used for extractor evaluation.

Silver Training Data

The silver training data was produced by:

1. 1.sampling papers from the ACL Anthology corpus
2. 2.generating synthetic questions from paper chunks
3. 3.rewriting those questions into retrieval-style queries
4. 4.retrieving top-ranked chunks
5. 5.annotating answer-bearing chunks with an LLM span extractor
6. 6.filtering noisy positives while preserving table/caption evidence

Intended Uses

• —Train query-conditioned token classifiers for semantic highlighting
• —Compare LLM teachers, token models, sentence-level compressors, and semantic highlighters under one span-scoring harness
• —Study boundary conventions for evidence extraction in scientific text

Limitations

• —The gold benchmark is small and single-annotator
• —Silver labels inherit retrieval noise and teacher noise
• —Tables and captions are represented through markdown/caption text, not full table structure
• —The benchmark reflects a paragraph-oriented annotation convention rather than a strict minimal answer-span convention

How To Load The Dataset

Load the canonical config:

from datasets import load_dataset

ds = load_dataset("KRLabsOrg/acl-verbatim-spans", "canonical")
train = ds["train"]
dev = ds["validation"]
test = ds["test"]

Load encoder-ready training rows:

from datasets import load_dataset

encoder = load_dataset("KRLabsOrg/acl-verbatim-spans", "encoder")
train = encoder["train"]
dev = encoder["validation"]

Example: inspect gold vs silver splits

from datasets import load_dataset

canonical = load_dataset("KRLabsOrg/acl-verbatim-spans", "canonical")

silver_train = canonical["train"]
silver_dev = canonical["validation"]
gold_test = canonical["test"]

Related Resources

• —Raw ACL Anthology markdown corpus: `KRLabsOrg/acl-anthology-md`
• —Codebase: `KRLabsOrg/acl-verbatim`

Citation

@misc{Recski:2026,
      title={ACL-Verbatim: hallucination-free question answering for research}, 
      author={Gábor Recski and Szilveszter Tóth and Nadia Verdha and István Boros and Ádám Kovács},
      year={2026},
      eprint={2605.21102},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.21102}, 
}