autores/imo_lq_filtered

Dataset Card

imolqfiltered

Dataset Details

We scraped conversations and their tags from topics posted on Art of Problem Solving's High School Olympiads section, then normalized the data and removed duplicates. We treated the first post in each topic as the Problem, and posts following it that potentially contained answers as Solutions.

Dataset Description

Using open-web-math/filtering-models, we removed text data with a perplexity greater than 15,000 in accordance with the paper's methodology. Additionally, for posts after the first one in each topic, we considered text data unlikely to be answers and removed it if it was 200 characters or less, or if its LaTeX ratio was 0.1 or lower.

Reference: https://huggingface.co/open-web-math/filtering-models/blob/main/example/perplexity.py

Dataset Sources

https://artofproblemsolving.com/community/c6h

Uses

We believe this dataset will be useful for fine-tuning language models on Mathematical Olympiad problems.