datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wikipedia-20230901.en-deduped
wikipedia - 20230901.en - deduped
purpose: train with less data while maintaining (most) of the quality
This is really more of a "high quality diverse sample" rather than "we are trying to remove literal duplicate documents". Source dataset: graelo/wikipedia.
configs
default
command:
python -m text_dedup.minhash \
--path $ds_name \
--name $dataset_config \
--split $data_split \
--cache_dir "./cache" \
--output $out_dir \
--column $text_column \… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/wikipedia-20230901.en-deduped.code_contests_instruct
Dataset Card for "code_contests_instruct"
The deepmind/code_contests dataset formatted as markdown-instruct for text generation training.
There are several different configs. Look at them. Comments:
flesch_reading_ease is computed on the description col via textstat
hq means that python2 (aka PYTHON in language column) is dropped, and keeps only rows with flesch_reading_ease 75 or greater
min-cols drops all cols except language and text
possible values for language are {'CPP'… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/code_contests_instruct.Long-Data-Col-rp_pile_pretrain
Dataset Card for "Long-Data-Col-rp_pile_pretrain"
This dataset is a subset of togethercomputer/Long-Data-Collections, namely the rp_sub.jsonl.zst and pile_sub.jsonl.zst files from the pretrain split.
Like the source dataset, we do not attempt to modify/change licenses of underlying data. Refer to the source dataset (and its source datasets) for details.
changes
as this is supposed to be a "long text dataset", we drop all rows where text contains <= 250 characters.… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/Long-Data-Col-rp_pile_pretrain.consumer-finance-complaints
BEE-spoke-data/consumer-finance-complaints
consumer-finance-complaints but in a format that actually works.
Pulled Feb 2024
TxT360-5M-sample-en
BEE-spoke-data/TxT360-5M-sample-en
english only sample from LLM360/TxT360:
min length 256 GPT-4 tokens
max length 24576 GPT-4 tokens
GPT-4 tiktoken token count:
token_count
count 5.000000e+06
mean 1.003614e+03
std 1.424231e+03
min 2.570000e+02
25% 4.020000e+02
50% 6.220000e+02
75% 1.050000e+03
max 2.457400e+04
Total count: 5018.07 M tokens
govdocs1-by-extension
govdocs1 Dataset: By File Extension
[!NOTE]
PDFs from govdocs1 are at this repo in "raw" file form - no simple "mostly correct" way to convert to text
Markdown-parsed versions of documents in govdocs1 with light filtering.
Usage
Load specific file formats (e.g., .doc files) parsed to markdown with pandoc:
from datasets import load_dataset
# Replace "doc" with desired config name
dataset = load_dataset("BEE-spoke-data/govdocs1-by-extension", "doc")… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-by-extension.UltraTextbooks-2.1-fw_mix
BEE-spoke-data/UltraTextbooks-2.1-fw_mix
filtered ultratextbooks for min 50 words
shuffle in 500k rows from fineweb to facilitate continual pretrain
GPT-4 tiktoken token count:
token_count
count 3.701646e+06
mean 9.934539e+02
std 1.726200e+03
min 5.400000e+01
25% 2.580000e+02
50% 5.540000e+02
75% 1.363000e+03
max 4.277600e+05
Total count: 3677.41 M tokens
TACO-hf
BEE-spoke-data/TACO-hf
Simple re-host of https://huggingface.co/datasets/BAAI/TACO but saved as hf dataset for ease of use.
Features:
DatasetDict({
"train": Dataset({
"features": [
"question",
"solutions",
"starter_code",
"input_output",
"difficulty",
"raw_tags",
"name",
"source",
"tags",
"skill_types",
"url",
"Expected Auxiliary… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/TACO-hf.reddit-title-body-hf
reddit-title-body-hf
sentence-transformers/reddit-title-body in parquet format
additional configs
the deduped config, which has the body col deduped via minhash
the mini config, which is a ~1 GB version of the deduped dataset created via a minipile-like clustering+sampling approach
fineweb-edu-10BT-mincols
fineweb-edu: 10BT sample
This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons.
token counts
GPT-4 tiktoken token count:
token_count
count 9.672101e+06
mean 1.001188e+03
std 1.834986e+03
min 3.800000e+01
25% 3.380000e+02
50% 6.090000e+02
75% 1.054000e+03
max 1.649670e+05
Total count: 9683.59 M tokens
cosmopedia-v2-mincols
cosmopedia-v2: mincols
cosmopedia-v2 with extra cols dropped to make the dataset smaller/easier to use
LONGCOT-merged-1Mthis is PowerInfer/QWQ-LONGCOT-500K + PowerInfer/LONGCOT-Refine-500K shuffled together with the following changes:
pointless starting phrases at the beginning (in english) such as "Okay, ..." "So, ..." etc are removed
config en has been filtered to include only rows detected as lang en in both prompt and response columns via fasttext
gutenberg-en-v1-clean
gutenberg - clean
dataset_info:
- config_name: default
features:
- name: text
dtype: string
- name: label
dtype: string
- name: score
dtype: float64
- name: sha256dtype: string
- name: word_count
dtype: int64
splits:
- name: train
num_bytes: 3384868097
num_examples: 9978
- name: validation
num_bytes: 195405579
num_examples: 574
- name: test
num_bytes: 189439446
num_examples: 565
download_size: 2317462261
dataset_size:… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/gutenberg-en-v1-clean.open-web-math-minhash
Dataset Card for "open-web-math-minhash"
An attempt at a "high quality sample" of open-web-math/open-web-math by aggressively applying minhash from text-dedup. The result is 1.82M rows down from the original 6M:
DatasetDict({
train: Dataset({
features: ['url', 'text', 'date', 'metadata'],
num_rows: 1820241
})
})
Usage
Unless you need the metadata, load the text-only config which is only 1.4 GB/5 shards:
from datasets import load_dataset… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/open-web-math-minhash.upvoteweb-posts
upvoteweb: posts
Posts in upvoteweb.
configs
[!IMPORTANT]There are several configs representing different permutations of this dataset. Load the relevant config for the task you are interested in.
Overview of configs:
default: largely unfiltered/unprocessed original data
eduscored: the "eduscore" predicted on the text column with huggingface's trained classifier
en-clean: filter language for en and language_score for > 0.6. Run clean-text on the text col, preserving… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/upvoteweb-posts.govdocs1-txt-raw
Dataset Card for "govdocs1-txt-raw"
Somewhere to put the raw txt files before filtering them
Source info/page: https://digitalcorpora.org/corpora/file-corpora/files/
@inproceedings{garfinkel2009bringing,
title={Bringing Science to Digital Forensics with Standardized Forensic Corpora},
author={Garfinkel, Simson and Farrell, Paul and Roussev, Vassil and Dinolt, George},
booktitle={Digital Forensic Research Workshop (DFRWS) 2009},
year={2009},
address={Montreal, Canada}… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-txt-raw.stackoverflow-questions-long
stackoverflow questions for text classification: 'long'
This is pacovaldez/stackoverflow-questions filtered for 1024 GPT2 tokens or more in title + body
https://huggingface.co/datasets/pacovaldez/stackoverflow-questions
napierone-epub-raw
BEE-spoke-data/napierone-epub-raw
NapierOne EPUB files converted with marker. Seems to contain mostly books from Project Gutenberg.
detected languages
via fasttext-langdetect
{'ca': 1,
'cy': 1,
'da': 6,
'de': 105,
'en': 4403,
'eo': 2,
'es': 61,
'fi': 76,
'fr': 189,
'he': 1,
'hu': 5,
'is': 1,
'it': 40,
'la': 6,
'nl': 41,
'pl': 4,
'pt': 38,
'sv': 10,
'tl': 9}
code-tutorials-en
Dataset Card for "code-tutorials-en"
en only
100 words or more
reading ease of 50 or more
DatasetDict({
train: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count', 'flesch_reading_ease'],
num_rows: 223162
})
validation: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count', 'flesch_reading_ease'],
num_rows: 5873
})
test: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count'… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/code-tutorials-en.financial-news-articles-filtereddataset_info:
features:
- name: title
dtype: string
- name: text
dtype: string
- name: url
dtype: string
- name: word_count
dtype: int64
splits:
- name: train
num_bytes: 554834105.9892601
num_examples: 199711
download_size: 459025008
dataset_size: 554834105.9892601
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
fineweb-1M_longish
fineweb "longish" 1M
1m samples w/ random seed w.r.t. previous samples.
min 512 GPT-4 tiktoken tokens
max 8192 GPT-4 tiktoken tokens
BEE-spoke-data/claude-tokenizer token count:
token_count
count 1000000.000000
mean 1218.231641
std 935.733312
min 139.000000
25% 683.000000
50% 905.000000
75% 1350.000000
max 9550.000000
Total count: 1218.23 M tokens
medium-articles-en
Dataset Card for "medium-articles-en"
fabiochiu/medium-articles filtered for en only and 100 GPT-4 tiktoken tokens or more.
TxT360-1M-sample
BEE-spoke-data/TxT360-1M-sample
One million row sample from LLM360/TxT360:
min length 256 GPT-4 tokens
max length 8192 GPT-4 tokens
rp_books-en
Dataset Card for "rp_books-en"
Filtering/cleaning on the 'red pajama books' subset of togethercomputer/Long-Data-Collections
The default config:
Dataset({
features: ['meta', 'text'],
num_rows: 26372
})
token count
default
GPT-4 tiktoken token count:
token_count
count 2.637200e+04
mean 1.009725e+05
std 1.161315e+05
min 3.811000e+03
25% 3.752750e+04
50% 7.757950e+04
75% 1.294130e+05
max 8.687685e+06
Total count: 2662.85 M… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/rp_books-en.SurvivorLib-rolmOCRAn initial version of the Survivor Library OCR'd via rolmOCR.
OCR of the books was completed by page, then joined together. You can split the text col on \f to get the pages for any given book
OCR of ancient books is not perfect - some pages have issues. We plan to attempt to detect these and fix/update over time
More details to come
Stats
Token Count
GPT-4 tiktoken token count:
token_count
count 1.461500e+04
mean 1.228607e+05
std 2.448000e+05
min… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/SurvivorLib-rolmOCR.awesome-python-apps
Dataset Card for "awesome-python-apps"
This contains .py files for the following repos taken from awesome-python-applications (on GitHub here)
abilian-sbe clone_repos.sh invesalius3 photonix sk1-wx
ambar CONTRIBUTING.md isso picard soundconverter
apatite CTFd kibitzrpi-hole soundgrain
ArchiveBox Cura KindleEar planet stargate… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/awesome-python-apps.napierone-pdf-nanonets-s
NapierOne PDFs: OCR'd by nanonets-s
PDFs from NapierOne (see 'pdf-total' in the napierone aws bucket) converted to text with nanonets-s using this code
contains results for all 4978 unique PDFs
raw config is unmodified from model output, the default config has been post-processed with mdformat
Citation
@article{DAVIES2022301330,
title = {NapierOne: A modern mixed file data set alternative to Govdocs1},
journal = {Forensic Science International: Digital… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/napierone-pdf-nanonets-s.napierone-pdf-olmOCR
NapierOne PDFs - converted with olmOCR
PDFs from NapierOne (see 'pdf-total' in the napierone aws bucket) converted to text via the olmOCR pipeline
outputs with <= 100 chars and 'low quality pdfs' were filtered out
the 4228 rows in default config represent approx 93,595 input PDF pages
Citation
@article{DAVIES2022301330,
title = {NapierOne: A modern mixed file data set alternative to Govdocs1},
journal = {Forensic Science International: Digital Investigation}… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/napierone-pdf-olmOCR.the-stack-smol-xs-all
bigcode/the-stack-smol-xs - all configs
All configs from bigcode/the-stack-smol-xs concatenated and shuffled. 100 examples each of:
['ada', 'agda', 'alloy', 'antlr', 'applescript', 'assembly', 'augeas', 'awk',
'batchfile', 'bison', 'bluespec', 'c', 'c++', 'c-sharp', 'clojure', 'cmake',
'coffeescript', 'common-lisp', 'css', 'cuda', 'dart', 'dockerfile', 'elixir',
'elm', 'emacs-lisp', 'erlang', 'f-sharp', 'fortran', 'glsl', 'go', 'groovy',
'haskell', 'html', 'idris', 'isabelle'… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/the-stack-smol-xs-all.fineweb-1M_en-med
fineweb 1M sample: 'medium' length
min 384 GPT4 tiktoken tokens
max 2048 GPT4 tiktoken tokens