indolem/IndoCareer

<p align="left"> <img src="https://raw.githubusercontent.com/fajri91/eval_picts/refs/heads/master/indocareer2.png" style="width: 80%;" id="title-icon"> </p>

Introduction

IndoCareer is a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. <p align="left"> <img src="https://raw.githubusercontent.com/fajri91/evalpicts/refs/heads/master/indocareerpie.png" style="width: 40%;" id="title-icon"> </p>

Data

Each question in the dataset is a multiple-choice question with up to 5 choices and only one choice as the correct answer.

import datasets
data = datasets.load_dataset('indolem/IndoCareer', 'all')

Examples

These questions are written in Indonesian.

<p align="left"> <img src="https://raw.githubusercontent.com/fajri91/evalpicts/refs/heads/master/indocareerexample.png" style="width: 40%;" id="title-icon">

</p>

Evaluation

We evaluated one closed-source model (GPT-4o) and 26 open-weight LLMs:

<p align="left"> <img src="https://raw.githubusercontent.com/fajri91/evalpicts/refs/heads/master/indocareerresult.png" style="width: 70%;" id="title-icon"> </p>

Citation

Please find out paper 📄<a href="https://arxiv.org/pdf/2409.08564" target="_blank" style="margin-right: 15px; margin-left: 10px">here.</a>

@inproceedings{koto2025cracking,
  title={Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia},
  author={"Fajri Koto"},
  booktitle={Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL HLT 2025), Industry Track},
  year={2025}
}