Datasetpublic

jkhouja/LingOly-TOO

LingOly-TOO (L2) Links 📊 Website and Leaderboard 📎 Paper 🧩 Code Summary LingOly-TOO (L2) is a challenging linguistics reasoning benchmark designed to counteracts answering without reasoning (e.g. by guessing or memorizing answers). Dataset format LingOly-TOO benchmark was created by generating up to 6 obfuscations per problem for 82 problems source from original LingOly benchmark. Dataset contains over 1200 question answer pairs. Some… See the full description on the dataset page: https://huggingface.co/datasets/jkhouja/LingOly-TOO.

sourceHugging Facecc-by-nc-nd-4.0updated 1y agoView on Hugging Face

2likes1.6kdownloads

Dataset Card

<div align="center"> <center><h1>LingOly-TOO (L2)</h1></center> </div>

[image]

Summary

LingOly-TOO (L2) is a challenging linguistics reasoning benchmark designed to counteracts answering without reasoning (e.g. by guessing or memorizing answers).

Dataset format

LingOly-TOO benchmark was created by generating up to 6 obfuscations per problem for 82 problems source from original LingOly benchmark. Dataset contains over 1200 question answer pairs. Some answers consists of multiple parts.

python

{'question_n':            # The question number in the problem
 'prompt':                # The main text of the question including preamble, context and previous questions
 'completion':            # The correct answer
 'question':              # The question text only (without the the rest of the prompt)
 'context':               # Context text that includes important information, you should prepend your prompt with context for solvability
 'obfuscated':            # If this example was obfuscated or not
 'overall_question_n':    # The problem number
 'obfuscated_question_n': # Concatenation of problem number and obfuscation number
}

Citation

@article{khouja2025lingolytoodisentanglingmemorisationreasoning,
      title={LINGOLY-TOO: Disentangling Memorisation from Reasoning with Linguistic Templatisation and Orthographic Obfuscation}, 
      author={Jude Khouja and Karolina Korgul and Simi Hellsten and Lingyi Yang and Vlad Neacsu and Harry Mayne and Ryan Kearns and Andrew Bean and Adam Mahdi},
      year={2025},
      eprint={2503.02972},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.02972}, 
}