datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
govdocs1-pdf-source
govdocs1: source PDF files
[!NOTE]
Converted versions of other document types (word, txt, etc) are available in this repo
This is ~220,000 open-access PDF documents (about 6.6M pages) from the dataset govdocs1. It wants to be OCR'd.
Uploaded as tar file pieces of ~10 GiB each due to size/file count limits with an index.csv covering details
5,000 randomly sampled PDFs are available unarchived in sample/. Hugging Face supports previewing these in-browser, for example this one… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-pdf-source.wikipedia-20230901.en-deduped
wikipedia - 20230901.en - deduped
purpose: train with less data while maintaining (most) of the quality
This is really more of a "high quality diverse sample" rather than "we are trying to remove literal duplicate documents". Source dataset: graelo/wikipedia.
configs
default
command:
python -m text_dedup.minhash \
--path $ds_name \
--name $dataset_config \
--split $data_split \
--cache_dir "./cache" \
--output $out_dir \
--column $text_column \… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/wikipedia-20230901.en-deduped.code_contests_instruct
Dataset Card for "code_contests_instruct"
The deepmind/code_contests dataset formatted as markdown-instruct for text generation training.
There are several different configs. Look at them. Comments:
flesch_reading_ease is computed on the description col via textstat
hq means that python2 (aka PYTHON in language column) is dropped, and keeps only rows with flesch_reading_ease 75 or greater
min-cols drops all cols except language and text
possible values for language are {'CPP'… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/code_contests_instruct.Long-Data-Col-rp_pile_pretrain
Dataset Card for "Long-Data-Col-rp_pile_pretrain"
This dataset is a subset of togethercomputer/Long-Data-Collections, namely the rp_sub.jsonl.zst and pile_sub.jsonl.zst files from the pretrain split.
Like the source dataset, we do not attempt to modify/change licenses of underlying data. Refer to the source dataset (and its source datasets) for details.
changes
as this is supposed to be a "long text dataset", we drop all rows where text contains <= 250 characters.… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/Long-Data-Col-rp_pile_pretrain.consumer-finance-complaints
BEE-spoke-data/consumer-finance-complaints
consumer-finance-complaints but in a format that actually works.
Pulled Feb 2024
TxT360-5M-sample-en
BEE-spoke-data/TxT360-5M-sample-en
english only sample from LLM360/TxT360:
min length 256 GPT-4 tokens
max length 24576 GPT-4 tokens
GPT-4 tiktoken token count:
token_count
count 5.000000e+06
mean 1.003614e+03
std 1.424231e+03
min 2.570000e+02
25% 4.020000e+02
50% 6.220000e+02
75% 1.050000e+03
max 2.457400e+04
Total count: 5018.07 M tokens
govdocs1-by-extension
govdocs1 Dataset: By File Extension
[!NOTE]
PDFs from govdocs1 are at this repo in "raw" file form - no simple "mostly correct" way to convert to text
Markdown-parsed versions of documents in govdocs1 with light filtering.
Usage
Load specific file formats (e.g., .doc files) parsed to markdown with pandoc:
from datasets import load_dataset
# Replace "doc" with desired config name
dataset = load_dataset("BEE-spoke-data/govdocs1-by-extension", "doc")… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-by-extension.UltraTextbooks-2.1-fw_mix
BEE-spoke-data/UltraTextbooks-2.1-fw_mix
filtered ultratextbooks for min 50 words
shuffle in 500k rows from fineweb to facilitate continual pretrain
GPT-4 tiktoken token count:
token_count
count 3.701646e+06
mean 9.934539e+02
std 1.726200e+03
min 5.400000e+01
25% 2.580000e+02
50% 5.540000e+02
75% 1.363000e+03
max 4.277600e+05
Total count: 3677.41 M tokens
TACO-hf
BEE-spoke-data/TACO-hf
Simple re-host of https://huggingface.co/datasets/BAAI/TACO but saved as hf dataset for ease of use.
Features:
DatasetDict({
"train": Dataset({
"features": [
"question",
"solutions",
"starter_code",
"input_output",
"difficulty",
"raw_tags",
"name",
"source",
"tags",
"skill_types",
"url",
"Expected Auxiliary… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/TACO-hf.bigpatent-all
BEE-spoke-data/bigpatent-all
Original bigpatent subset "all" converted to hf format, columns renamed, some cleaning applied to "summary" column.
additional configs
the deduped config, which has the summary col deduped via minhash
the mini config, which is a ~1 GB version of the deduped dataset created via a minipile-like clustering+sampling approach
flan-v2-hf
flan-v2: hf datasets format
https://hf.co/datasets/philschmid/flanv2 directly loaded into hf datasets parquet files for easier streaming, etc.
[!NOTE]
There are two other configs besides the default, en-targets and targets-3w
en-targets: the targets column filtered with fasttext-langdetect for en and score >= 0.65
targets-3w: the targets column filtered for 3 or more words
reddit-title-body-hf
reddit-title-body-hf
sentence-transformers/reddit-title-body in parquet format
additional configs
the deduped config, which has the body col deduped via minhash
the mini config, which is a ~1 GB version of the deduped dataset created via a minipile-like clustering+sampling approach
fineweb-edu-10BT-mincols
fineweb-edu: 10BT sample
This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons.
token counts
GPT-4 tiktoken token count:
token_count
count 9.672101e+06
mean 1.001188e+03
std 1.834986e+03
min 3.800000e+01
25% 3.380000e+02
50% 6.090000e+02
75% 1.054000e+03
max 1.649670e+05
Total count: 9683.59 M tokens
cosmopedia-v2-mincols
cosmopedia-v2: mincols
cosmopedia-v2 with extra cols dropped to make the dataset smaller/easier to use
FLAN-compressed-plusplus
flan-compressed ++
FLAN-compressed with additional tasks added, mostly programming-related. As per Google's flan-v2 README, they seemingly excluded most (all?) programming/code tasks from the data they published.
README WIP
bees
Dataset Card for Bees
Dataset Summary
The USNM Bumblebee Dataset is a natural history dataset containing, for each of 73,497 Bumblebee specimens in the family Apidae, a single image in lateral or dorsal view and a tab-separated value file with occurrence data. Occurrence data includes the species classification, the date and site/location of collection, and other metadata conforming to the Darwin Core data standard (https://dwc.tdwg.org). 11,421 specimens are not… See the full description on the dataset page: https://huggingface.co/datasets/MikeTrizna/bees.LONGCOT-merged-1Mthis is PowerInfer/QWQ-LONGCOT-500K + PowerInfer/LONGCOT-Refine-500K shuffled together with the following changes:
pointless starting phrases at the beginning (in english) such as "Okay, ..." "So, ..." etc are removed
config en has been filtered to include only rows detected as lang en in both prompt and response columns via fasttext
gutenberg-en-v1-clean
gutenberg - clean
dataset_info:
- config_name: default
features:
- name: text
dtype: string
- name: label
dtype: string
- name: score
dtype: float64
- name: sha256dtype: string
- name: word_count
dtype: int64
splits:
- name: train
num_bytes: 3384868097
num_examples: 9978
- name: validation
num_bytes: 195405579
num_examples: 574
- name: test
num_bytes: 189439446
num_examples: 565
download_size: 2317462261
dataset_size:… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/gutenberg-en-v1-clean.open-web-math-minhash
Dataset Card for "open-web-math-minhash"
An attempt at a "high quality sample" of open-web-math/open-web-math by aggressively applying minhash from text-dedup. The result is 1.82M rows down from the original 6M:
DatasetDict({
train: Dataset({
features: ['url', 'text', 'date', 'metadata'],
num_rows: 1820241
})
})
Usage
Unless you need the metadata, load the text-only config which is only 1.4 GB/5 shards:
from datasets import load_dataset… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/open-web-math-minhash.v3-mix-mixtralgovdocs1-image
BEE-spoke-data/govdocs1-image
This contains .jpg files from govdocs1. Light deduplication was applied (i.e. jdupes on all files) which removed ~500 duplicate images.
DatasetDict({
train: Dataset({
features: ['image'],
num_rows: 108895
})
})
source
Source info/page: https://digitalcorpora.org/corpora/file-corpora/files/
@inproceedings{garfinkel2009bringing,
title={Bringing Science to Digital Forensics with Standardized Forensic Corpora}… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-image.upvoteweb-posts
upvoteweb: posts
Posts in upvoteweb.
configs
[!IMPORTANT]There are several configs representing different permutations of this dataset. Load the relevant config for the task you are interested in.
Overview of configs:
default: largely unfiltered/unprocessed original data
eduscored: the "eduscore" predicted on the text column with huggingface's trained classifier
en-clean: filter language for en and language_score for > 0.6. Run clean-text on the text col, preserving… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/upvoteweb-posts.govdocs1-txt-raw
Dataset Card for "govdocs1-txt-raw"
Somewhere to put the raw txt files before filtering them
Source info/page: https://digitalcorpora.org/corpora/file-corpora/files/
@inproceedings{garfinkel2009bringing,
title={Bringing Science to Digital Forensics with Standardized Forensic Corpora},
author={Garfinkel, Simson and Farrell, Paul and Roussev, Vassil and Dinolt, George},
booktitle={Digital Forensic Research Workshop (DFRWS) 2009},
year={2009},
address={Montreal, Canada}… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-txt-raw.stackoverflow-questions-long
stackoverflow questions for text classification: 'long'
This is pacovaldez/stackoverflow-questions filtered for 1024 GPT2 tokens or more in title + body
https://huggingface.co/datasets/pacovaldez/stackoverflow-questions
napierone-epub-raw
BEE-spoke-data/napierone-epub-raw
NapierOne EPUB files converted with marker. Seems to contain mostly books from Project Gutenberg.
detected languages
via fasttext-langdetect
{'ca': 1,
'cy': 1,
'da': 6,
'de': 105,
'en': 4403,
'eo': 2,
'es': 61,
'fi': 76,
'fr': 189,
'he': 1,
'hu': 5,
'is': 1,
'it': 40,
'la': 6,
'nl': 41,
'pl': 4,
'pt': 38,
'sv': 10,
'tl': 9}
code-tutorials-en
Dataset Card for "code-tutorials-en"
en only
100 words or more
reading ease of 50 or more
DatasetDict({
train: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count', 'flesch_reading_ease'],
num_rows: 223162
})
validation: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count', 'flesch_reading_ease'],
num_rows: 5873
})
test: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count'… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/code-tutorials-en.financial-news-articles-filtereddataset_info:
features:
- name: title
dtype: string
- name: text
dtype: string
- name: url
dtype: string
- name: word_count
dtype: int64
splits:
- name: train
num_bytes: 554834105.9892601
num_examples: 199711
download_size: 459025008
dataset_size: 554834105.9892601
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
FLAN-compressed
FLAN: compressed
An aggressively deduplicated (aka "compressed") version of FLAN: 378M rows -> 116M
deduplicated on inputs only with minhash
all rows mentioning translation tasks in _task_name have been dropped (see config all for these included)
Top 40 tasks - Default Config - Click to Expand
Top 40 tasks - Default Config:
[('gigaword:1.2.0', 7225415),
('amazon_polarity_flattering_or_not', 6966639),
('amazon_polarity_Is_this_review_negative', 6912683)… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/FLAN-compressed.fineweb-1M_longish
fineweb "longish" 1M
1m samples w/ random seed w.r.t. previous samples.
min 512 GPT-4 tiktoken tokens
max 8192 GPT-4 tiktoken tokens
BEE-spoke-data/claude-tokenizer token count:
token_count
count 1000000.000000
mean 1218.231641
std 935.733312
min 139.000000
25% 683.000000
50% 905.000000
75% 1350.000000
max 9550.000000
Total count: 1218.23 M tokens
medium-articles-en
Dataset Card for "medium-articles-en"
fabiochiu/medium-articles filtered for en only and 100 GPT-4 tiktoken tokens or more.