datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
LongBench-v2
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
🌐 Project Page: https://longbench2.github.io
💻 Github Repo: https://github.com/THUDM/LongBench
📚 Arxiv Paper: https://arxiv.org/abs/2412.15204
LongBench v2 is designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. LongBench v2 has the following features: (1) Length: Context length ranging from 8k to… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/LongBench-v2.LongAlign-10k
LongAlign-10k
🤗 [LongAlign Dataset] • 💻 [Github Repo] • 📃 [LongAlign Paper]
LongAlign is the first full recipe for LLM alignment on long context. We propose the LongAlign-10k dataset, containing 10,000 long instruction data of 8k-64k in length. We investigate on trianing strategies, namely packing (with loss weighting) and sorted batching, which are all implemented in our code. For real-world long context evaluation, we introduce LongBench-Chat that evaluate the… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/LongAlign-10k.LongCite-45k
LongCite-45k
🤗 [LongCite Dataset] • 💻 [Github Repo] • 📃 [LongCite Paper]
LongCite-45k dataset contains 44,600 long-context QA instances paired with sentence-level citations (both English and Chinese, up to 128,000 words). The data can support training long-context LLMs to generate response and fine-grained citations within a single output.
Data Example
Each instance in LongCite-45k consists of an instruction, a long context (divided into sentences), a user… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/LongCite-45k.webglm-qa
WebGLM-QA
Dataset Description
WebGLM-QA is the dataset used to train the WebGLM generator module. It consists of 43,579 high-quality data samples for the train split, 1,000 for the validation split, and 400 for the test split. Refer to our paper for the data construction details.
Dataset Structure
To load the dataset, you can try the following code.
from datasets import load_dataset
load_dataset("THUDM/webglm-qa")
DatasetDict({
train: Dataset({… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/webglm-qa.AISE-Bench
AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs
🌐 Project Page •
💻 GitHub •
📖 KDD 2026 Paper
AISE-Bench is a real-world benchmark for information seeking on academic knowledge graphs. It is built from authentic AMiner user search queries and provides human-verified academic question-answering data with executable multi-step API trajectories, standardized tool inputs, API execution outputs, and… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/AISE-Bench.Qwen_mmlu_dataset Cleaned MMLU Dataset for Reliable Multitask Evaluation
This dataset is a cleaned and deduplicated version of the original MMLU (Massive Multitask Language Understanding) benchmark, curated to ensure high-quality, model-ready samples for fine-tuning and evaluation.
Key Improvements:
Removed duplicate entries and inconsistent formatting
Validated answer keys and standardized subject labels
Ensured uniform structure across all splits for seamless integration
Use Cases:
- Fine-tuning… See the full description on the dataset page: https://huggingface.co/datasets/zail-ai/Qwen_mmlu_dataset.