datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-ultra-mini-pro
Dataset Card for Fineweb Ultra Mini
Fineweb Ultra Mini is a dataset derived from the original Fineweb dataset made by huggingface (see here: https://huggingface.co/datasets/HuggingFaceFW/fineweb).
The dataset focuses on extracting high quality data from the Fineweb dataset, from the 1-0.5% range. If you would like more data, though slightly sacrificing quality check out fineweb ultra mini, which focuses on the 2-3% of high quality data originally found in fineweb.… See the full description on the dataset page: https://huggingface.co/datasets/motionlabs/fineweb-ultra-mini-pro.fineweb_CC-MAIN-2024-18_100k_output_UncovAI_83362
What is it?
As more and more data are generated daily, It becomes important to be able to distinguish between synthetic and human data for model training.
We analyzed the first 100k rows of the Fineweb dataset focusing on the dump CC-MAIN-2024-18 using the UncovAI model for text.
We observed that more than 16% of the data were detected as having been generated by AI by our model. We removed them and obtained a dataset of 83362 lines with a number of token approaching 55 million.… See the full description on the dataset page: https://huggingface.co/datasets/UncovAI/fineweb_CC-MAIN-2024-18_100k_output_UncovAI_83362.