LLaMAX/BenchMAX_Multiple_Functions

Dataset Sources

• —Paper: BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models
• —Link: https://huggingface.co/papers/2502.07346
• —Repository: https://github.com/CONE-MT/BenchMAX

Dataset Description

BenchMAXMultipleFunctions is a dataset of BenchMAX, sourcing from Nexus. This dataset evaluates the tool use capability in multilingual senarios, which requires a model to call the correct function given the user query and multiple functions.

We translate the standardized queries from English to 16 non-English languages by google Translate. Some special function arguments remain English since the APIs are in English. All samples are post-edited by native speakers.

Usage

git clone https://github.com/CONE-MT/BenchMAX.git
cd BenchMAX
pip install -r requirements.txt

cd tasks/nexus
languages=(en ar bn cs de es fr hu ja ko ru sr sw te th vi zh)
for lang in "${languages[@]}"; do
    python evaluator.py -m ${model} --infer-backend vllm -t ${lang} --output-parser-name generic
done

Supported Languages

Arabic, Bengali, Chinese, Czech, English, French, German, Hungarian, Japanese, Korean, Serbian, Spanish, Swahili, Telugu, Thai, Russian, Vietnamese

Citation

If you find our dataset helpful, please cite this paper:

@article{huang2025benchmax,
  title={BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models},
  author={Huang, Xu and Zhu, Wenhao and Hu, Hanxu and He, Conghui and Li, Lei and Huang, Shujian and Yuan, Fei},
  journal={arXiv preprint arXiv:2502.07346},
  year={2025}
}