quality-filter
DUE-v2-quality-filtered
removed markdown formatting, links, emojis, multiple white spaces
replaced user and channel tags with @user and #channel
deduplicated (not fuzzy only exact)
basic filtering:
word count filter(min_words=3, max_words=200)
mean word length filter(min_mean_word_length=2.0, max_mean_word_length=12.0)
long word filter(max_word_length=200)
whitespace filter (max_white_space_ratio=0.25)
Qwen2.5-3B-Paper-Quality-Filter-Datasetolympiads_math_220k_filtered_single_gen_high_qualitylerobot_datasets_corn_high_quality_50_filter_15lerobot_datasets_corn_high_quality_50_filter_15_sorted_cropedlerobot_datasets_corn_high_quality_50_sorted_filter_15