datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wikipedia-20230901.en-deduped
wikipedia - 20230901.en - deduped
purpose: train with less data while maintaining (most) of the quality
This is really more of a "high quality diverse sample" rather than "we are trying to remove literal duplicate documents". Source dataset: graelo/wikipedia.
configs
default
command:
python -m text_dedup.minhash \
--path $ds_name \
--name $dataset_config \
--split $data_split \
--cache_dir "./cache" \
--output $out_dir \
--column $text_column \… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/wikipedia-20230901.en-deduped.code_contests_instruct
Dataset Card for "code_contests_instruct"
The deepmind/code_contests dataset formatted as markdown-instruct for text generation training.
There are several different configs. Look at them. Comments:
flesch_reading_ease is computed on the description col via textstat
hq means that python2 (aka PYTHON in language column) is dropped, and keeps only rows with flesch_reading_ease 75 or greater
min-cols drops all cols except language and text
possible values for language are {'CPP'… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/code_contests_instruct.Long-Data-Col-rp_pile_pretrain
Dataset Card for "Long-Data-Col-rp_pile_pretrain"
This dataset is a subset of togethercomputer/Long-Data-Collections, namely the rp_sub.jsonl.zst and pile_sub.jsonl.zst files from the pretrain split.
Like the source dataset, we do not attempt to modify/change licenses of underlying data. Refer to the source dataset (and its source datasets) for details.
changes
as this is supposed to be a "long text dataset", we drop all rows where text contains <= 250 characters.… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/Long-Data-Col-rp_pile_pretrain.consumer-finance-complaints
BEE-spoke-data/consumer-finance-complaints
consumer-finance-complaints but in a format that actually works.
Pulled Feb 2024
TxT360-5M-sample-en
BEE-spoke-data/TxT360-5M-sample-en
english only sample from LLM360/TxT360:
min length 256 GPT-4 tokens
max length 24576 GPT-4 tokens
GPT-4 tiktoken token count:
token_count
count 5.000000e+06
mean 1.003614e+03
std 1.424231e+03
min 2.570000e+02
25% 4.020000e+02
50% 6.220000e+02
75% 1.050000e+03
max 2.457400e+04
Total count: 5018.07 M tokens
UltraTextbooks-2.1-fw_mix
BEE-spoke-data/UltraTextbooks-2.1-fw_mix
filtered ultratextbooks for min 50 words
shuffle in 500k rows from fineweb to facilitate continual pretrain
GPT-4 tiktoken token count:
token_count
count 3.701646e+06
mean 9.934539e+02
std 1.726200e+03
min 5.400000e+01
25% 2.580000e+02
50% 5.540000e+02
75% 1.363000e+03
max 4.277600e+05
Total count: 3677.41 M tokens
govdocs1-by-extension
govdocs1 Dataset: By File Extension
[!NOTE]
PDFs from govdocs1 are at this repo in "raw" file form - no simple "mostly correct" way to convert to text
Markdown-parsed versions of documents in govdocs1 with light filtering.
Usage
Load specific file formats (e.g., .doc files) parsed to markdown with pandoc:
from datasets import load_dataset
# Replace "doc" with desired config name
dataset = load_dataset("BEE-spoke-data/govdocs1-by-extension", "doc")… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-by-extension.TACO-hf
BEE-spoke-data/TACO-hf
Simple re-host of https://huggingface.co/datasets/BAAI/TACO but saved as hf dataset for ease of use.
Features:
DatasetDict({
"train": Dataset({
"features": [
"question",
"solutions",
"starter_code",
"input_output",
"difficulty",
"raw_tags",
"name",
"source",
"tags",
"skill_types",
"url",
"Expected Auxiliary… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/TACO-hf.reddit-title-body-hf
reddit-title-body-hf
sentence-transformers/reddit-title-body in parquet format
additional configs
the deduped config, which has the body col deduped via minhash
the mini config, which is a ~1 GB version of the deduped dataset created via a minipile-like clustering+sampling approach
fineweb-edu-10BT-mincols
fineweb-edu: 10BT sample
This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons.
token counts
GPT-4 tiktoken token count:
token_count
count 9.672101e+06
mean 1.001188e+03
std 1.834986e+03
min 3.800000e+01
25% 3.380000e+02
50% 6.090000e+02
75% 1.054000e+03
max 1.649670e+05
Total count: 9683.59 M tokens
cosmopedia-v2-mincols
cosmopedia-v2: mincols
cosmopedia-v2 with extra cols dropped to make the dataset smaller/easier to use
sponsorblock-youtube-metadata-2024
SponsorBlock YouTube Metadata Dataset
A dataset of YouTube video metadata collected from a subset of videos in the SponsorBlock database. This dataset contains metadata, subtitles, engagement heatmaps, live chat, and channel playlist information for popular YouTube videos.
Contains the top videos from the SponsorBlock database that had data added in the year 2024.
Quick Stats
Metric
Value
Total videos
154,536
Videos with subtitles
62,819 (41%)… See the full description on the dataset page: https://huggingface.co/datasets/ScriptSmith/sponsorblock-youtube-metadata-2024.IndustryCorpus_sports[中文主页]
Industry models play a crucial role in driving enterprise intelligence transformation and innovative development. High-quality industry data is key to improving the performance of large models and realizing industry applications. However, datasets currently used for industry model training generally suffer from issues such as insufficient data volume, low quality, and lack of domain expertise.
To address these problems, we constructed and applied 22 industry data processing operators to… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus_sports.SpotifyLyrics001LONGCOT-merged-1Mthis is PowerInfer/QWQ-LONGCOT-500K + PowerInfer/LONGCOT-Refine-500K shuffled together with the following changes:
pointless starting phrases at the beginning (in english) such as "Okay, ..." "So, ..." etc are removed
config en has been filtered to include only rows detected as lang en in both prompt and response columns via fasttext
spoken-magpie-ja
Spoken-magpie
LLMの日本語Instruction Tuning用データllm-jp/magpie-sft-v1.0をCosyVoice2 TTSを使用して音声化した商用利用可能な日本語の音声言語モデルのSFT用データセットです。
ある程度の話者多様性を持つように生成されています。
Respone Audioは500文字以下の場合にのみ生成されています。
NVIDIA H200を10枚を使用しvllmで推論しました。
Samples
最初の50サンプルを掲載します。
ID
Instruction
Instruction Audio
Response
Response Audio
0
カボチャを使ったスイーツのレシピをいくつか教えてください。
もちろんです、カボチャを使ったスイーツは秋にぴったりですね。以下にいくつかのレシピをご紹介します。1. カボチャのスフレパウンドケーキ- 材料：カボチャ 200g、生クリーム 50ml、牛乳 50ml、卵 3個、砂糖 100g、薄力粉 70g、バニラエッセンス 少々-… See the full description on the dataset page: https://huggingface.co/datasets/Atotti/spoken-magpie-ja.open-web-math-minhash
Dataset Card for "open-web-math-minhash"
An attempt at a "high quality sample" of open-web-math/open-web-math by aggressively applying minhash from text-dedup. The result is 1.82M rows down from the original 6M:
DatasetDict({
train: Dataset({
features: ['url', 'text', 'date', 'metadata'],
num_rows: 1820241
})
})
Usage
Unless you need the metadata, load the text-only config which is only 1.4 GB/5 shards:
from datasets import load_dataset… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/open-web-math-minhash.gutenberg-en-v1-clean
gutenberg - clean
dataset_info:
- config_name: default
features:
- name: text
dtype: string
- name: label
dtype: string
- name: score
dtype: float64
- name: sha256dtype: string
- name: word_count
dtype: int64
splits:
- name: train
num_bytes: 3384868097
num_examples: 9978
- name: validation
num_bytes: 195405579
num_examples: 574
- name: test
num_bytes: 189439446
num_examples: 565
download_size: 2317462261
dataset_size:… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/gutenberg-en-v1-clean.upvoteweb-posts
upvoteweb: posts
Posts in upvoteweb.
configs
[!IMPORTANT]There are several configs representing different permutations of this dataset. Load the relevant config for the task you are interested in.
Overview of configs:
default: largely unfiltered/unprocessed original data
eduscored: the "eduscore" predicted on the text column with huggingface's trained classifier
en-clean: filter language for en and language_score for > 0.6. Run clean-text on the text col, preserving… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/upvoteweb-posts.govdocs1-txt-raw
Dataset Card for "govdocs1-txt-raw"
Somewhere to put the raw txt files before filtering them
Source info/page: https://digitalcorpora.org/corpora/file-corpora/files/
@inproceedings{garfinkel2009bringing,
title={Bringing Science to Digital Forensics with Standardized Forensic Corpora},
author={Garfinkel, Simson and Farrell, Paul and Roussev, Vassil and Dinolt, George},
booktitle={Digital Forensic Research Workshop (DFRWS) 2009},
year={2009},
address={Montreal, Canada}… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-txt-raw.stackoverflow-questions-long
stackoverflow questions for text classification: 'long'
This is pacovaldez/stackoverflow-questions filtered for 1024 GPT2 tokens or more in title + body
https://huggingface.co/datasets/pacovaldez/stackoverflow-questions
qwen3.5-2b-base-blind-spots
Qwen3.5-2B-Base — Blind Spot Analysis (Text + Vision)
Model Tested
Field
Value
Model
Qwen/Qwen3.5-2B-Base
Parameters
2.27 B (2,274 M per HF metadata)
Architecture
Hybrid Gated-DeltaNet (dense FFN) — 24 LM layers (18 DeltaNet + 6 full-attention), ViT vision encoder
Type
Pre-trained base model (not instruction-tuned)
Context
262 144 tokens
Modalities
Text + Vision (early-fusion multimodal)
Key Contributions
Only multimodal… See the full description on the dataset page: https://huggingface.co/datasets/F555/qwen3.5-2b-base-blind-spots.napierone-epub-raw
BEE-spoke-data/napierone-epub-raw
NapierOne EPUB files converted with marker. Seems to contain mostly books from Project Gutenberg.
detected languages
via fasttext-langdetect
{'ca': 1,
'cy': 1,
'da': 6,
'de': 105,
'en': 4403,
'eo': 2,
'es': 61,
'fi': 76,
'fr': 189,
'he': 1,
'hu': 5,
'is': 1,
'it': 40,
'la': 6,
'nl': 41,
'pl': 4,
'pt': 38,
'sv': 10,
'tl': 9}
financial-news-articles-filtereddataset_info:
features:
- name: title
dtype: string
- name: text
dtype: string
- name: url
dtype: string
- name: word_count
dtype: int64
splits:
- name: train
num_bytes: 554834105.9892601
num_examples: 199711
download_size: 459025008
dataset_size: 554834105.9892601
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
code-tutorials-en
Dataset Card for "code-tutorials-en"
en only
100 words or more
reading ease of 50 or more
DatasetDict({
train: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count', 'flesch_reading_ease'],
num_rows: 223162
})
validation: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count', 'flesch_reading_ease'],
num_rows: 5873
})
test: Dataset({
features: ['text', 'url', 'dump', 'source', 'word_count'… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/code-tutorials-en.bprna-spot
bpRNA-spot
bpRNA-spot is a collection of the datasets used by SPOT-RNA for RNA secondary structure prediction.
The dataset is released as a composite repository, bpRNA-spot, and three numbered component repositories:
bpRNA-spot-0: the initial bpRNA split, TR0, VL0, and TS0.
bpRNA-spot-1: the PDB transfer-learning split, TR1, VL1, and TS1.
bpRNA-spot-2: the NMR-only evaluation split, TS2.
bpRNA-spot concatenates the components in order:
train: TR0 + TR1
validation: VL0 + VL1
test:… See the full description on the dataset page: https://huggingface.co/datasets/multimolecule/bprna-spot.fineweb-1M_longish
fineweb "longish" 1M
1m samples w/ random seed w.r.t. previous samples.
min 512 GPT-4 tiktoken tokens
max 8192 GPT-4 tiktoken tokens
BEE-spoke-data/claude-tokenizer token count:
token_count
count 1000000.000000
mean 1218.231641
std 935.733312
min 139.000000
25% 683.000000
50% 905.000000
75% 1350.000000
max 9550.000000
Total count: 1218.23 M tokens
TxT360-1M-sample
BEE-spoke-data/TxT360-1M-sample
One million row sample from LLM360/TxT360:
min length 256 GPT-4 tokens
max length 8192 GPT-4 tokens
task362_spolin_yesand_prompt_response_sub_classification
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task362_spolin_yesand_prompt_response_sub_classification
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task362_spolin_yesand_prompt_response_sub_classification.SportsMetrics
SportsMetrics
Benchmark data to evaluate numerical reasoning and information fusion of LLMs.
SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Dong Yu, Fei Liu In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL'24), Bangkok, Thailand. Arxiv Paper
Usage
from datasets import load_dataset
def get_task(domain… See the full description on the dataset page: https://huggingface.co/datasets/huuuyeah/SportsMetrics.