ameyhengle/Multilingual-Needle-in-a-Haystack
Multilingual Needle in a Haystack (MLNeedle) The MultiLingual Needle-in-a-Haystack (MLNeedle) test is a dataset designed to assess how well Large Language Models (LLMs) find specific information ("needle") within long, multilingual texts ("haystack"). Built on MLQA, it contains over 5,000 extractive question-answer instances across seven languages (English, Arabic, German, Spanish, Hindi, Vietnamese, Simplified Chinese). We systematically vary the "needle's" language and… See the full description on the dataset page: https://huggingface.co/datasets/ameyhengle/Multilingual-Needle-in-a-Haystack.
Multilingual Needle in a Haystack (MLNeedle)
The MultiLingual Needle-in-a-Haystack (MLNeedle) test is a dataset designed to assess how well Large Language Models (LLMs) find specific information ("needle") within long, multilingual texts ("haystack"). Built on MLQA, it contains over 5,000 extractive question-answer instances across seven languages (English, Arabic, German, Spanish, Hindi, Vietnamese, Simplified Chinese). We systematically vary the "needle's" language and position to evaluate LLM robustness in long-context scenarios.
Dataset Structure
The MLNeedle dataset is organized into configurations based on the total context length, mirroring the experimental setups in the original paper. Each configuration (e.g., 4k, 8k, 16k, 32k) contains splits for each of the seven languages, indicating the language of the 'needle' document.
How to Load
You can load the MLNeedle dataset using the Hugging Face datasets library. You need to specify the config_name for the desired context length and the split for the desired needle language.
from datasets import load_dataset
# Load the 4k context length configuration with English needles
dataset_4k_en = load_dataset("ameyhengle/Multilingual-Needle-in-a-Haystack", "4k", split="en")
print(dataset_4k_en)
# Load the 8k context length configuration with Spanish needles
dataset_8k_es = load_dataset("ameyhengle/Multilingual-Needle-in-a-Haystack", "8k", split="es")
print(dataset_8k_es)
# To iterate through all available configurations and splits
for config in ["4k", "8k", "16k", "32k"]:
for lang in ["en", "es", "de", "zh", "vi", "ar", "hi"]:
dataset = load_dataset("ameyhengle/Multilingual-Needle-in-a-Haystack", config, split=lang)
print(f"Loaded config: {config}, language: {lang}, size: {len(dataset)}")Citation
If you use the MLNeedle dataset in your research, please cite the original paper:
@inproceedings{hengle-etal-2025-multilingual,
title = "Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models",
author = "Hengle, Amey and
Bajpai, Prasoon and
Dan, Soham and
Chakraborty, Tanmoy",
editor = "Chiruzzo, Luis and
Ritter, Alan and
Wang, Lu",
booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.naacl-long.267/",
doi = "10.18653/v1/2025.naacl-long.267",
pages = "5165--5180",
ISBN = "979-8-89176-189-6",
}