iapp/openai_humaneval-th

HumanEval-th

A Thai translation of all 164 problems of OpenAI's HumanEval. Every row corresponds 1:1, in order, to a row of the English original, so the Thai and English scores of a model are directly comparable.

Only the prompt column is Thai. canonical_solution, test and entry_point are Python rather than prose and were never translated; they are byte-identical to openai/openai_humaneval in all 164 rows, and validate.py checks that on every run.

Every revision of this dataset before August 2026 was unusable. The file was double-JSON-escaped: 4,537 literal backslash-n sequences, not one real newline, so every program was a single unbroken line and all 164 prompts failed ast.parse. If you have a score computed on an earlier revision, it does not measure what you think it measures. See CHANGELOG.md.

Source and licence

Translation provenance

Machine translation, with no human post-editing, followed by a repair pass in August 2026 over the rows where that showed. What was wrong and what was done about it is set out in `CHANGELOG.md`.

The pre-escaping output of the original translation run no longer exists. The repair therefore worked from the published file plus the English original, and CHANGELOG.md states which of the two each change came from.

Fields

Verifying this card

validate.py checks every claim above against the live English original. It needs pandas and pyarrow:

python validate.py

• —Structural — 164 rows; task_id in the English order; canonical_solution, test and entry_point byte-identical to openai/openai_humaneval; every prompt parses as Python. Non-zero exit if any of these fail.
• —Fidelity — per-row translation quality: no doctest example of the English original lost, no code line glued to a docstring delimiter, no doctest marker left without an example after it, every docstring closed on its own line, every prompt containing Thai. Reported as counts; --strict makes them fail the run too.

At the current revision both tiers pass, 6/6 and 5/5, so validate.py --strict exits 0.

A note on HumanEval/51

Its worked example and its test both contain the Python source literal "abcdef\nghijklm", where backslash-n has to stay two characters because it is the thing being tested. The double-escaping decoded it into a real line break, and no process working from the damaged file alone can tell that case apart from a line break that was always a line break. Both were restored from the English original. This is the only place in the dataset where a value was taken from the English rather than repaired in place, and it restores a column that was never translated to begin with.

Citation

@article{chen2021codex,
  title   = {Evaluating Large Language Models Trained on Code},
  author  = {Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and others},
  journal = {arXiv preprint arXiv:2107.03374},
  year    = {2021}
}

Thai translation by iApp Technology.