ruibrogandrade/ARC-Challenge_PT-PT

Dataset Card for ARC-Challenge_PT-PT

Dataset Summary

This repository contains a European Portuguese (pt-PT) translation of ARC-Challenge (AI2 Reasoning Challenge – Challenge subset), a benchmark of non-trivial, grade-school science questions that require background knowledge and reasoning. Each example presents a question and four answer choices. Load with:

import datasets
data = datasets.load_dataset("ruibrogandrade/ARC-Challenge_PT-PT")

Supported Tasks and Leaderboards

• —Multiple-Choice Question Answering (scientific commonsense)
• —Choice selection / classification (accuracy)

Languages

• —European Portuguese (pt-PT) for translated fields
• —English originals are included for reference

Example

An example from the test split:

{
  "id": "Mercury_SC_415702",
  "question": "George wants to warm his hands quickly by rubbing them. Which skin surface will produce the most heat?",
  "choices": {
    "text": ["dry palms", "wet palms", "palms covered with oil", "palms covered with lotion"],
    "label": ["A","B","C","D"]
  },
  "answerKey": "A",
  "question_translated": "O George quer aquecer rapidamente as mãos esfregando-as. Que superfície da pele produzirá mais calor?",
  "choices_translated": {
    "text": ["palmas secas", "palmas húmidas", "palmas cobertas de óleo", "palmas cobertas com loção"],
    "label": ["A","B","C","D"]
  }
}

Dataset Structure

• —id (string): item identifier.
• —question (string): science question (EN).
• —choices (dict): options with text (list of strings) and label (list of option letters).
• —answerKey (string): gold answer letter (available on test).
• —question_translated (string): pt-PT question.
• —choices_translated (dict): pt-PT options (parallel to choices).

Data Splits

• —Train: 1,095 examples
• —Test: 979 examples
• —Validation: not included

Dataset Creation

Curation & pipeline. Translated with TowerInstruct-13B using question/choice-specific prompts and beam search. Post-processing. Cleaning removed artefacts/prefixes, normalized punctuation/whitespace, and enforced structural consistency (e.g., ensuring answer choice labels remain letters A–D; rows with numeric labels were discarded). Nested list/dict fields were cleaned element-wise.

Additional Information

License: Same as the original ARC dataset; see the source dataset’s license terms.

Citation

@article{clark2018think,
  title   = {Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge},
  author  = {Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind},
  journal = {arXiv preprint arXiv:1803.05457},
  year    = {2018}
}

@mastersthesis{andrade2025,
  title  = {Benchmarking Portuguese via Machine Translation and LLMs},
  author = {Andrade, Rui Brogueira},
  school = {Faculdade de Engenharia da Universidade do Porto (FEUP)},
  year   = {2025},
  url    = {https://repositorio-aberto.up.pt/handle/10216/168804}
}