ZhikaiXu24/DAS-Bench

DAS-Bench

DAS-Bench is a 30-topic multi-domain benchmark for automatically generated academic surveys. It is paired with DAS-Eval, a 16-criterion evaluation suite for publication-oriented academic surveys covering scholarly citation, taxonomic synthesis, hierarchical discourse, and manuscript reliability.

• —Paper: Deep Academic Survey
• —Project page: DAS
• —Source and evaluation toolkit: ZhikaiXu24/DAS
• —Literature metadata lake: ZhikaiXu24/DAS-2M

This repository is the Hugging Face distribution of the current public DAS-Bench release. The GitHub repository remains the canonical source for the evaluation implementation and development history.

Load with datasets

from datasets import load_dataset

topics = load_dataset(
    "ZhikaiXu24/DAS-Bench",
    "topics",
    split="benchmark",
)

results_30 = load_dataset(
    "ZhikaiXu24/DAS-Bench",
    "results_30_topics",
    split="evaluation",
)

results_cs21 = load_dataset(
    "ZhikaiXu24/DAS-Bench",
    "results_cs21",
    split="evaluation",
)

Dataset configurations

The matched CS subset is not a separate benchmark split. It contains the 21 computer science topics shared by every evaluated system and supports comparison under identical topic coverage.

Current release scope

The current public release includes:

• —the frozen 30-topic list;
• —the benchmark task definition and metadata schema;
• —the BSC, MAR, TSQ, and HDQ metric definitions;
• —the DAS-Eval implementation and safe placeholder configuration;
• —the public aggregate results; and
• —qualitative comparison materials.

This release does not include the fixed 300-paper candidate pool for each topic, survey-generation code, formal submission-format documentation, private experiment artifacts, or model outputs. Candidate pools may be added in a future release without changing the current topics and results configurations.

Benchmark task

Each complete benchmark instance defines a research topic and a fixed pool of 300 candidate-paper metadata records. The 30 topics span language models, computer vision, robotics, scientific discovery, security, medicine, climate, remote sensing, finance, and uncertainty estimation.

See `benchmark/task_specification.md` for task boundaries and `benchmark/metadata_schema.json` for the candidate-paper metadata schema.

DAS-Eval

DAS-Eval defines four complementary metric families, each containing four criteria scored from 1 to 5:

• —Balanced Scholarly Citation Quality (BSC)
• —Manuscript Artifact Reliability (MAR)
• —Taxonomic Synthesis Quality (TSQ)
• —Hierarchical Drafting Quality (HDQ)

The reported Total Avg. is the arithmetic mean of all 16 criteria. Complete definitions and aggregation rules are provided in `benchmark/evaluation_protocol.md`.

Evaluation toolkit

The public toolkit is available under `evaluation/`:

• —eval_prepare.py: prepares Markdown and rendered page images from survey PDFs;
• —eval_bsc.py: evaluates BSC;
• —eval_mar.py: evaluates MAR;
• —eval_tsq_hdq.py: evaluates TSQ and HDQ; and
• —run_eval_all.sh: runs the evaluation stages serially.

Install the common dependencies with:

python -m pip install -r requirements.txt

Configure the hosted or local OpenAI-compatible judge in config.json. API credentials must be supplied through environment variables and must never be committed. PDF preprocessing uses MinerU, which must be installed separately.

Qualitative comparison

Download the original qualitative case study PDF.

[image]

License

The files in this repository are released under the Apache License 2.0. DAS-2M is distributed separately under the terms on its dataset page.

Citation

@article{xu2026deepacademicsurvey,
  title   = {Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation},
  author  = {Xu, Zhikai and Xue, Zhucun and Hu, Teng and Wang, Yabiao and Liu, Yong and Zhang, Jiangning},
  journal = {arXiv preprint},
  year    = {2026}
}