datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-ultra-mini
Dataset Card for Fineweb Ultra Mini
Fineweb Ultra Mini is a dataset derived from the original Fineweb dataset made by huggingface (see here: https://huggingface.co/datasets/HuggingFaceFW/fineweb).
The dataset focuses on extracting high quality data from the Fineweb dataset, from the 2-3% range. If you would like even more high-quality data, keep out for our next release, fineweb ultra mini pro, which focuses on the 0-1% of high quality data originally found in fineweb.… See the full description on the dataset page: https://huggingface.co/datasets/motionlabs/fineweb-ultra-mini.Par-Four-Fineweb-Edu-FortifiedDataset Summary
This dataset is a filtered subset of the Fineweb-Edu-Fortified dataset. The primary goal of this subset is to reduce the dataset size to a more manageable volume while maintaining high-quality content. It contains three key fields: score, text, and url, focusing on entries with a score of 4 and above, indicating higher relevance and quality of educational content.
This dataset can be used for several fine-tuning and model improvement tasks, including model healing, synthetic… See the full description on the dataset page: https://huggingface.co/datasets/Josephgflowers/Par-Four-Fineweb-Edu-Fortified.Par-Four-Fineweb-Edu-Fortified-Chemistry-Physics-Astronomy-Math-ReasonExtracted Chemistry Physics Asronomy Math and Logic portions from the original.
Script used for the extraction:
https://huggingface.co/datasets/Josephgflowers/Par-Four-Fineweb-Edu-Fortified-Chemistry-Physics-Astronomy-Math-Reason/resolve/main/find-science-fine.py
FineWebUA-links-p1-500m
Intro
This dataset contains links to Ukrainian websites and sitemaps derived from the Common Crawl corpus. A part of the Fineweb-UA project.
Stats
42263126 part-00000.sitemap-links.tsv
45051846 part-00001.sitemap-links.tsv
26360778 part-00002.sitemap-links.tsv
49879673 part-00003.sitemap-links.tsv
29744936 part-00004.sitemap-links.tsv
61139341 part-00005.sitemap-links.tsv
43183739 part-00006.sitemap-links.tsv
53279243 part-00007.sitemap-links.tsv
60319695… See the full description on the dataset page: https://huggingface.co/datasets/Yehor/FineWebUA-links-p1-500m.fineweb_synthetic_queriesPar-Four-Fineweb-Edu-Fortified-FinanceSubset of https://huggingface.co/datasets/Josephgflowers/Par-Four-Fineweb-Edu-Fortified
Used keyword filtering and scoring to extract data with a finance focus.
Creation script:
https://huggingface.co/datasets/Josephgflowers/Par-Four-Fineweb-Edu-Fortified-Finance/resolve/main/find-fin-fine.py
Dataset Description
Summary
This dataset is a finance-focused filtered subset of the Fineweb-Edu-Fortified dataset. It is designed to extract and prioritize high-quality… See the full description on the dataset page: https://huggingface.co/datasets/Josephgflowers/Par-Four-Fineweb-Edu-Fortified-Finance.finewebedu-sentences
Fineweb-edu Sentences
Description:
A dataset of sentences collected from the web.
The dataset was created by splitting the text into individual sentences using the spaCy package,
then removing duplicates and filtering for complete sentences in a semi-automated process.
Source: HuggingFaceFW/fineweb-edu
Size: About 700,000 English language sentences. Each sentence is 512 tokens long or less as assessed using the BERT tokenizer.
Annotations: The source field contains the URL of each… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/finewebedu-sentences.fineweb-ultra-mini-pro
Dataset Card for Fineweb Ultra Mini
Fineweb Ultra Mini is a dataset derived from the original Fineweb dataset made by huggingface (see here: https://huggingface.co/datasets/HuggingFaceFW/fineweb).
The dataset focuses on extracting high quality data from the Fineweb dataset, from the 1-0.5% range. If you would like more data, though slightly sacrificing quality check out fineweb ultra mini, which focuses on the 2-3% of high quality data originally found in fineweb.… See the full description on the dataset page: https://huggingface.co/datasets/motionlabs/fineweb-ultra-mini-pro.Par-Four-Fineweb-Edu-Fortified-LogicFiltered for logic with the following scrip:
https://huggingface.co/datasets/Josephgflowers/Par-Four-Fineweb-Edu-Fortified-Logic/resolve/main/find-reason-fine.py
Par-Four-Fineweb-Edu-Fortified-MathFiltered for a math focus.
Script used to create the dataset:
https://huggingface.co/datasets/Josephgflowers/Par-Four-Fineweb-Edu-Fortified-Math/resolve/main/find-math-fine.py
FineWeb-alphanum
FineWeb-alphanum
FineWeb-Edu-25Kfinewebs-copyright-domains
List of domains that were removed from FineWeb(-2)
An easy-access dataset that contains the domains that were removed from FineWeb(-2) after take-down requests. This should allow others to quickly filter out domains that should not be included in data collection, removing redundant data processing. If used in this way, as a domain-wide filter, it should make life easier for the copyright holders, too, who will not have to resubmit cease-and-desists for each new webcrawl.
Currently… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/finewebs-copyright-domains.fineweb-edu-gemini-annotations-portuguese-regressionfineweb-afifllm-5gib-csv
Generated and extracted data to train the AfifLLM_Hyper_2
extracted from:https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle
FineWeb-Edu_10B_sample_2_column_word_countsThis dataset contains is a 2-column csv file representation of the FineWeb-Edu 10B sample dataset which is made available under the Open Data Commons Attribution License (ODC-By). All the texts in the ~27GB of parquet files were split according to the regex below using the Python regex package (not re).
The text chunks from these splits were counted to make the str: int mapping of the csv file. The result is a greater than 100X reduction in file size (27GB -> 240MB) making it easy to fit this… See the full description on the dataset page: https://huggingface.co/datasets/alexandermorgan/FineWeb-Edu_10B_sample_2_column_word_counts.fineweb_CC-MAIN-2024-18_100k_output_UncovAI_83362
What is it?
As more and more data are generated daily, It becomes important to be able to distinguish between synthetic and human data for model training.
We analyzed the first 100k rows of the Fineweb dataset focusing on the dump CC-MAIN-2024-18 using the UncovAI model for text.
We observed that more than 16% of the data were detected as having been generated by AI by our model. We removed them and obtained a dataset of 83362 lines with a number of token approaching 55 million.… See the full description on the dataset page: https://huggingface.co/datasets/UncovAI/fineweb_CC-MAIN-2024-18_100k_output_UncovAI_83362.fineweb2_ar_15m_sample