opencompass/NeedleBench

1---2license: mit3configs:4- config_name: multi_needle_reasoning_needle5  data_files:6  - split: test7    path:8    - multi_needle_reasoning_zh.json9    - multi_needle_reasoning_en.json10- config_name: zh_haystack_texts11  data_files:12  - split: test13    path:14    - zh_finance.jsonl15    - zh_game.jsonl16    - zh_general.jsonl17    - zh_government.jsonl18    - zh_movie.jsonl19    - zh_tech.jsonl20- config_name: en_haystack_texts21  data_files:22  - split: test23    path:24    - PaulGrahamEssays.jsonl25- config_name: atc_needles26  data_files:27  - split: test28    path:29    - names.json30- config_name: retrieval_needles31  data_files:32  - split: test33    path:34    - needles.jsonl35task_categories:36- question-answering37---38 39# Dataset Description40 41## Dataset Summary42 43The NeedleBench dataset is a part of the OpenCompass project, designed to evaluate the capabilities of large language models (LLMs) in processing and understanding long documents. It includes a series of test scenarios that assess models' abilities in long text information extraction and reasoning. The dataset is structured to support tasks such as single-needle retrieval, multi-needle retrieval, multi-needle reasoning, and ancestral trace challenges.44 45<div style="text-align: center;">46    <img src="https://github.com/user-attachments/assets/b895e0cf-4307-47d8-8e5a-9a4d1c58fa37" alt="Needlebench Overview" width="900" style="margin: auto;">47</div>48 49## Supported Tasks and Primary Languages50 51- **Single-Needle Retrieval Task (S-RT)**: Extracting a single key piece of information from a long text.52- **Multi-Needle Retrieval Task (M-RT)**: Retrieving multiple related pieces of information from long texts.53- **Multi-Needle Reasoning Task (M-RS)**: Extracting and utilizing multiple key pieces of information for comprehensive understanding.54- **Ancestral Trace Challenge (ATC)**: Handling multi-layer logical challenges in real long texts.55 56The dataset supports multiple languages, including English and Chinese, as indicated by the presence of files like `multi_needle_reasoning_en.json` and `multi_needle_reasoning_zh.json`.57 58## Potential Use Cases59 60The NeedleBench dataset can be used to evaluate and compare the performance of different large language models in tasks involving long text processing, information extraction, and reasoning. It is useful for researchers and developers working on models that need to handle complex queries on extensive documents.61 62## Evaluation63 64Please follow the provided guidelines in the [OpenCompass documentation](https://opencompass.readthedocs.io/en/latest/advanced_guides/needleinahaystack_eval.html) to set up the environment, configure the dataset, and run evaluations.65 66## Additional Information67 68For more details on the dataset, please refer to the [NeedleBench Technical Report](https://arxiv.org/abs/2407.11963).69 70Github repository: https://github.com/open-compass/opencompass71 72## Contact73 74For any questions or issues related to the dataset, please contact the maintainers or contributors of the [OpenCompass project](https://github.com/open-compass/opencompass).75 76## Citation77 78If you use this dataset, please add a reference:79 80```81@misc{li2024needlebenchllmsretrievalreasoning,82      title={NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context Window?},83      author={Mo Li and Songyang Zhang and Yunxin Liu and Kai Chen},84      year={2024},85      eprint={2407.11963},86      archivePrefix={arXiv},87      primaryClass={cs.CL},88      url={https://arxiv.org/abs/2407.11963},89}90```