datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SlimPajama-6BSampled version of cerebras/SlimPajama-627B.
Since the original data was shuffled before chunking, I only downloaded train/chunk1 (of 10 total) and further sampled 10%. This should result in roughly 6B tokens, hence SlimPajama-6B.
The dataset is 24GBs in storage size when decompressed (original dataset is over 2TBs) and has 5489000 rows.
The validation set and test set were sampled as well.
Data source proportions for SlimPajama-627B and SlimPajama-6B
For sanity purpose, I… See the full description on the dataset page: https://huggingface.co/datasets/DKYoon/SlimPajama-6B.SlimPajama-627B_ReuploadAs datasets puts limits on the number of calls to huggingface, downloading SlimPajama-627B is problematic as it's composed of a ton of small files.
I have reuploaded it here as larger chunks to easily download the dataset without having to do anything hacky.
The original dataset can be found here https://huggingface.co/datasets/cerebras/SlimPajama-627B
SlimPajama-Meta-rater
Annotated SlimPajama Dataset
Dataset Description
This dataset contains the first fully annotated SlimPajama dataset with comprehensive quality metrics for data-centric large language model research. The dataset includes approximately 580 billion tokens from the training set of the original SlimPajama dataset, annotated across 25 different quality dimensions.
Note: This dataset contains only the training set portion of the original SlimPajama dataset, which is why the… See the full description on the dataset page: https://huggingface.co/datasets/opendatalab/SlimPajama-Meta-rater.slimetaoshite300nenshiranaiuchinilevelmaxninattemashitasononi
Bangumi Image Base of Slime Taoshite 300-nen, Shiranai Uchi Ni Level Max Ni Nattemashita: Sono Ni
This is the image base of bangumi Slime Taoshite 300-nen, Shiranai Uchi ni Level Max ni Nattemashita: Sono Ni, we detected 66 characters, 7416 images in total. The full dataset is here.
Please note that these image bases are not guaranteed to be 100% cleaned, they may be noisy actual. If you intend to manually train models using this dataset, we recommend performing necessary… See the full description on the dataset page: https://huggingface.co/datasets/BangumiBase/slimetaoshite300nenshiranaiuchinilevelmaxninattemashitasononi.SlimOrca
Overview
This is a new curated subset of our OpenOrca data. This release provides an efficient means of reaching performance on-par with using larger slices of our data, while only including ~500k GPT-4 completions.
The key change in this dataset is that we've done an additional pass, using GPT-4 to remove answers which appear wrong based on the human annotations from the FLAN dataset.
This reduces the dataset size to only ~500k entries, allowing training to a similar quality level… See the full description on the dataset page: https://huggingface.co/datasets/Open-Orca/SlimOrca.slimpajama_Llama2_Tokenizer
slimpajama_Llama2_Tokenizer
The original slimpajama_Llama2_Tokenizer.tar.gz archive (≈794 GB on Ubuntu) was split into smaller 40 GB chunks for easier upload to Hugging Face.
sudo apt install git-lfs
pip install -U huggingface_hub # `hf version`==1.1.4
tar cvf - slimpajama_Llama2_Tokenizer/ | pigz -p 16 > slimpajama_Llama2_Tokenizer.tar.gz
split -b 40G -d -a 3 slimpajama_Llama2_Tokenizer.tar.gz slimpajama_Llama2_Tokenizer/slimpajama_Llama2_Tokenizer_part_
# Upload files… See the full description on the dataset page: https://huggingface.co/datasets/jsun/slimpajama_Llama2_Tokenizer.Slim-binidxThis dataset comprises nine chunks (out of ten) from the Cerebras/SlimPajama-627B dataset, processed into a binary index (bin idx) format.
The first chunk is located at : rwkv-x-dev/slimpajama-binidx.
Due to their large size, each chunk is split into multiple parts for easier handling. To reassemble and decompress these parts, follow these steps:
Combine all parts of the desired chunk into a single file:
cat chunk2_text_document_part_* > chunk2_text_document.tar.xz
Decompress the combined… See the full description on the dataset page: https://huggingface.co/datasets/something-else/Slim-binidx.SlimOrcaDedupCleaned
What is this dataset?
Half of the Slim Orca Deduped dataset, but further cleaned by removing instances of soft prompting.
I removed a ton prompt prefixes which did not add any information or were redundant. Ex. "Question:", "Q:", "Write the Answer:", "Read this:", "Instructions:"
I also removed a ton of prompt suffixes which were simply there to lead the model to answer as expected Ex. "The answer is...", "Answer:", "A:", "Summary:", "Output:", "Highlight:"
Why?
I… See the full description on the dataset page: https://huggingface.co/datasets/cgato/SlimOrcaDedupCleaned.slimder-qwen38-ngram-activation-refine-20260901
REAM-288 activation-aware 50% PLE selection
This dataset contains the verified capacity-preserving row selection and remap
tables for compacting the PLE n-gram memory in
sjakek/slimder-qwen38-ream288-depth32-agentic to exactly 50% of its padded
physical capacity.
The selection combines a 250M-token multi-domain frequency census with a
3M-token proportional-stratified PLE activation-saliency pass. The saliency
sample contains 900k code, 750k agentic, 450k retrieval, 300k… See the full description on the dataset page: https://huggingface.co/datasets/jakeatx/slimder-qwen38-ngram-activation-refine-20260901.SlimOrca-Dedup
Overview
"SlimOrca Dedup" is a deduplicated, unfiltered subset of the SlimOrca dataset, excluding RLHF instances, resulting in 363k unique examples.
Key Features
Removal of RLHF instances.
Deduplication using minhash and Jaccard similarity techniques.
Demo Models
Note: These models were trained on the full SlimOrca dataset, not the deduplicated, unfiltered version.
* https://huggingface.co/openaccess-ai-collective/jackalope-7b
*… See the full description on the dataset page: https://huggingface.co/datasets/Open-Orca/SlimOrca-Dedup.slimpajama-per-source-length-upsampleSlimPajama-6B_km-ip-d512SlimPajama-chunked
SlimPajama-Chunked
Dataset Description
This is a chunked re-upload of Cerebras' SlimPajama-627B. The original upload has split
the dataset into 10 chunks, with each containing upwards of 5,000 files. This makes it cumbersome to download and process. We've downloaded the entire
dataset for our own purposes, and decided to upload the chunked version for easier usage.
Each file is ~45GB due to HuggingFace's limitation of 50GB per LFS file.
SlimPajama_300BThe SlimPajama_300B is a 300B token sample of de-duplicated Slim Pajama dataset tokenized using the EleutherAI/gpt-neox-20b tokenizer
Due to file size constraints, C4 and CommonCrawl has been uploaded in multiple chunks, you can use the following commands to merge them back into a single file:
cat C4_part_* > C4.bin
cat CommonCrawl_part_* > CommonCrawl.bin
Data Distribution
Data source
Composition
Commoncrawl
0.5208
C4
0.2668
GitHub
0.0522
Books
0.0420
ArXiv… See the full description on the dataset page: https://huggingface.co/datasets/SpectraSuite/SlimPajama_300B.SlimPajama-6B-processed-8192slimpajama-6B_qwen3-8B_embedSlimPajama-15bslimpajama_dc_lc_mistralSlimPajama-3Bslimpajama_labeledThis is the dataset used in the paper Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining.
It is a labeled version of the SlimPajama-627B train dataset.
An example in this dataset:
{
"id": "BkiUdvk25V5jCITp144_",
"content": "At the time of Federation most Australian colonies had introduced income taxes, each with its own rules and administered in its own way. This was further complicated with some jurisdictions recording tax according to a taxpayer's residence, and… See the full description on the dataset page: https://huggingface.co/datasets/beccabai/slimpajama_labeled.SlimPajama-15bSlimPajama-627B_ReuploadAs datasets puts limits on the number of calls to huggingface, downloading SlimPajama-627B is problematic as it's composed of a ton of small files.
I have reuploaded it here as larger chunks to easily download the dataset without having to do anything hacky.
The original dataset can be found here https://huggingface.co/datasets/cerebras/SlimPajama-627B
slimder-qwen38-condensed-agentic100-20260901slimetaoshite300nenshiranaiuchinilevelmaxninattemashita
Bangumi Image Base of Slime Taoshite 300-nen, Shiranai Uchi Ni Level Max Ni Nattemashita
This is the image base of bangumi Slime Taoshite 300-nen, Shiranai Uchi ni Level Max ni Nattemashita, we detected 68 characters, 6333 images in total. The full dataset is here.
Please note that these image bases are not guaranteed to be 100% cleaned, they may be noisy actual. If you intend to manually train models using this dataset, we recommend performing necessary preprocessing on the… See the full description on the dataset page: https://huggingface.co/datasets/BangumiBase/slimetaoshite300nenshiranaiuchinilevelmaxninattemashita.SlimOrca
Dataset Card for Isotonic/SlimOrca
Dataset Summary
This dataset is a deduplicated version of Open-Orca/OpenOrca
MinHash Deduplication with Jaccard Threshold = 0.80
Original dataset size: 4233923
Number of duplicate clusters: 522077
Files in duplicate cluster: 2115143
Unique files in duplicate cluster: 892638
Filtered dataset size: 3011418
Whisper_Slim_MegiJiRack-SlimOrca_4k-Datasetslimpajama_dc_lc_llamaicp_slimpajamaSLiMIA
SLiMIA - Spheroid Light Microscopy Image Atlas
Binary segmentation of 3D tumour and non-tumour spheroids in transmitted-light
microscopy, spanning 9 microscopes, 48 cell lines, 8 culture media and 4 spheroid
formation methods. Mirror of figshare collection
7486311.
License
CC BY 4.0 - all nine figshare data articles carry it (verified against the
figshare API license field per article and against DataCite SPDX).
Note the trap: the accompanying Scientific Data… See the full description on the dataset page: https://huggingface.co/datasets/MedOtter/SLiMIA.