datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
lls-filtering-datadatafilteringnetworks_2b
Getting Started
To use the indices, first download CommonPool: https://github.com/mlfoundations/datacomp#downloading-commonpool
Then reshard with the provided indices file: https://github.com/mlfoundations/datacomp#selecting-samples-in-the-filtering-track
Paper Link
https://arxiv.org/abs/2309.17425
instruction_filtering_askllm_seed_data_code_w_openthoughtsinstruction_filtering_embedding_filter_seed_data_math_w_openthoughtsfiltering-reference-datadata-filtering-statisticsFilterings on top of near-dedup + line filtering:
Comments filtering (at least 1% of the number of lines should be comments/docstrings)
Stars filtering (minimum of 5 stars) (on top of near-dedup + line filtering)
Language
Before filtering
Stars
Comments ratio
More near-dedup
Tokenizer fertility
Python
75.61 GB
26.56 GB
65.64 GB
61.97 GB
72.52 GB
Java
110 GB
35.83 GB
92.7 GB
88.42 GB
105.47 GB
Javascript
82.7 GB
20.76 GB
57.5 GB
65.09 GB
76.37 GB
filtering_data-v11instruction_filtering_askllm_seed_data_math_w_openthoughtsinstruction_filtering_embedding_filter_mean_seed_data_math_w_openthoughtsinstruction_filtering_gemini_length_seed_data_code_w_openthoughtsinstruction_filtering_gemini_length_seed_data_math_w_openthoughtsinstruction_filtering_fasttext_per_domain_seed_data_code_w_openthoughtsinstruction_filtering_fasttext_per_domain_seed_data_math_w_openthoughtsinstruction_filtering_embedding_filter_mean_seed_data_code_w_openthoughtsinstruction_filtering_embedding_filter_seed_data_code_w_openthoughtsinstruction_filtering_fasttext_seed_data_code_w_openthoughtsinstruction_filtering_random_seed_data_code_w_openthoughtsinstruction_filtering_fasttext_seed_data_math_w_openthoughtsinstruction_filtering_random_seed_data_math_w_openthoughtsfiltering_data-v112instruction_filtering_fasttext_seed_datainstruction_filtering_fasttext_seed_data_mathinstruction_filtering_fasttext_seed_data_codeinstruction_filtering_fasttext_per_domain_seed_data_mathinstruction_filtering_fasttext_per_domain_seed_data_codeinstruction_filtering_askllm_seed_data_mathinstruction_filtering_askllm_seed_data_codeMT_DataFiltering_Stage1