Tanushreeeeee/CodeMixBench

ℹ️Dataset Card for CodeMixBench

[EMNLP'25] CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages

<a href="https://github.com/Jeromeyluck/CodeMixBench" target="_blank"> <img alt="Github" src="https://img.shields.io/badge/🐙-Github-blue" /> </a>

<a href="https://arxiv.org/abs/2507.18791" target="blank"> <img alt="Paper" src="https://img.shields.io/badge/📜-Paper-purple" /> </a> <a href="https://2025.emnlp.org/" target="blank"> <img alt="EMNLP 2025" src="https://img.shields.io/badge/Proceedings-EMNLP2025-blue" /> </a>

<!-- Provide a quick summary of the dataset. -->

Code-mixing is a linguistic phenomenon where multilingual speakers switch or mix two or more languages within a single utterance or conversation. To evaluate LLMs’ comprehension of multilingual code-mixed texts, we introduce CodeMixBench, a benchmark comprising eight tasks across 18 languages.

[image]

🔎Dataset Details

Our benchmark comprises synthesized datasets targeting knowledge reasoning, mathematical reasoning, and truthfulness tasks, along with LID, POS, NER, SA, and MT tasks, which have been adapted from open-source studies.

CodeMixBench vs. Others

Previous benchmarks, such as GLUECoS and LinCE, primarily focus on traditional NLP tasks and are limited to a small number of languages. LinCE includes four language pairs and five NLP tasks: Language Identification(LID), Part of Speech (POS), Named Entity Recognition (NER), Sentiment Analysis (SA), and Machine Translation (MT). In contrast, GLUECoS covers two language pairs, lacks the MT task, but adds Question Answering (QA) and Natural Language Inference (NLI). Our review of recent codemixing studies indicates that research extends beyond the language pairs used in LinCE and GLUECoS. Therefore, we expanded to 16 language pairs and introduced tasks better suited for evaluating LLMs, such as Multi-Choice, Math, and Truthfulness, resulting in a total of eight tasks.

[image]

Statistics of Synthetic Datasets

For knowledge reasoning, we developed the code-mixed MMLU (CM-MMLU) based on the MMLU test set, featuring multiple-choice questions from 57 subjects to assess the model's comprehensive knowledge reasoning abilities. For mathematical reasoning, we created the code-mixed GSM8K (CM-GSM8K), derived from the GSM8K test set, which evaluates mathematical reasoning capabilities with each question including step-by-step solutions. For truthfulness assessment, we constructed the code-mixed TruthfulQA (CM-TruthfulQA) using 817 multiple-choice questions from the TruthfulQA test set.

[image]

Statistics of Collected Datasets

We selected and reconstructed 30 datasets from existing open-source projects. To comprehensively evaluate the performance of large models on code-mixing, we aimed to encompass a diverse range of language families and tasks, prioritizing manually annotated datasets. Ultimately, we cover traditional NLP tasks such as Language Identification (LID), Named Entity Recognition (NER), Part-of-Speech tagging (POS), Sentiment Analysis(SA), and Machine Translation (MT), and cover 16 languages from seven language families: Germanic(en, de, nl, fy), Sino-Tibetan (zh, hok), Romance(es), Afro-Asiatic (msa, ea), Indo-Aryan (hi, bn, ne,mr), Dravidian (ta, ml), and Tupian (gn).

[image]

Experience Results

We evaluate three families of LLMs on CodeMixBench, revealing consistent underperformance across all models on code-mixing datasets involving language pairs from different language families. However, enhancements in training data size, model scale, post-training, and few-shot learning can improve LLM performance on code-mixing datasets.

[image]

[image]

🚀Load CodeMixBench

Taking the GSM8K task with mixed Chinese and English, gsm8k_chieng, as an example.

   from datasets import load_dataset

   dataset_dict = load_dataset('CodeMixBench/CodeMixBench', data_files={'test': './gsm8k/gsm8k_chieng.csv'})

📍Dataset Sources

<!-- Provide the basic links for the dataset. -->

• —Repository: https://github.com/Jeromeyluck/CodeMixBench/
• —Paper: CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages

Setup

1. 1.Follow these steps to set up your development environment:

   git clone git@github.com:Jeromeyluck/CodeMixBench.git
   cd CodeMixBench

   conda create -n CodeMixBench python=3.9
   conda activate CodeMixBench
   pip install -r requirements.txt

1. 1.To launch an llm for testing:

   python ./test_model.py \
     --dataset lid_guaspa \
     --expid lid_guaspa_all_0shot \
     --model gpt-3.5-turbo \
     --shot 5 \
     --api sk-********************* \
     --url https://****************

• —dataset: select the dataset (e.g., lid_gereng, lid_spaeng, ner_hineng).
• —expid: define the ID of the test, the results file will be named after this ID.
• —model: the model you test. The default model is gpt-3.5-turbo.
• —shot: use for few-shot test (by default it will be 1).
• —api: API Key (default key will be OPENAI_API_KEY defined in system path).
• —url: API function provider's URL.

🔗Citation

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

BibTeX:

 @misc{yang2025codemixbenchevaluatingcodemixingcapabilities,
    title={CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages}, 
    author={Yilun Yang and Yekun Chai},
    year={2025},
    eprint={2507.18791},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2507.18791}, 
}