datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
State-Parse-FilteredThe single cell RNA-seq dataset with human PBMC samples was sourced from Parse Biosciences [1]. [1] Performance of Evercode™ WT v3 in Human Immune Cells (PBMCs), https://www.parsebiosciences.com/datasets/performance-of-evercode-wt-v3-in-human-immune-cells-pbmcs/; Parse Biosciences, Seattle, USA; accessed 05/27/2025.
Certain uses of this data may require a license from Parse Biosciences, Inc.
OmniEdit-Filtered-1.2M
OmniEdit
In this paper, we present OMNI-EDIT, which is an omnipotent editor to handle seven different image editing tasks with any aspect ratio seamlessly. Our contribution is in four folds: (1) OMNI-EDIT is trained by utilizing the supervision
from seven different specialist models to ensure task coverage. (2) we utilize importance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score to improve the data quality.
📃Paper | 🌐Website |… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/OmniEdit-Filtered-1.2M.self-oss-instruct-sc2-exec-filter-50kFinal self-alignment training dataset for StarCoder2-Instruct.
seed: Contains the seed Python function
concepts: Contains the concepts generated from the seed
instruction: Contains the instruction generated from the concepts
response: Contains the execution-validated response to the instruction
This dataset utilizes seed Python functions derived from the MultiPL-T pipeline.
CodeFeedback-Filtered-Instruction OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
[🏠Homepage]
|
[🛠️Code]
OpenCodeInterpreter
OpenCodeInterpreter is a family of open-source code generation systems designed to bridge the gap between large language models and advanced proprietary systems like the GPT-4 Code Interpreter. It significantly advances code generation capabilities by integrating execution and iterative refinement functionalities.
For further information and… See the full description on the dataset page: https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction.MultiLegalPile_Wikipedia_FilteredA filtered version of the MultiLegalPile dataset, together with wikipedia articles.hoigen-filtered-videos
HOIGen Filtered Videos Dataset
This dataset contains 28562 filtered videos from the HOIGen-1M dataset based on the allowlist.
Dataset Structure
The videos are organized in the same structure as the original HOIGen dataset:
filtered_videos/
├── videos_part_1/
├── videos_part_2/
├── ...
└── videos_part_100/
Usage
from huggingface_hub import hf_hub_download
# Download a specific video
video_path = hf_hub_download(… See the full description on the dataset page: https://huggingface.co/datasets/charlychan123/hoigen-filtered-videos.objaverse-filtered
3D Model Dataset (built from allenai/objaverse)
Filtered, Blender-validated subset of allenai/objaverse.
Model files are preserved byte-for-byte from the source; each record carries
Blender-extracted geometry, materials, textures, scene structure, and hierarchy.
Layout
data/shard-XXXXXX/models/ original model files ({uid}.glb)
data/shard-XXXXXX/metadata.jsonl one record per model
data/shard-XXXXXX/train.jsonl chat-format LLM fine-tuning pairs… See the full description on the dataset page: https://huggingface.co/datasets/raj2708/objaverse-filtered.esb-datasets-earnings22-validation-tiny-filteredA filtered (<=30s duration) slice (512 samples) of the Earnings22 dataset.
def add_duration(sample):
y, sr = sample['audio']["array"], sample['audio']["sampling_rate"]
sample['duration_ms']=librosa.get_duration(y=y, sr=sr) * 1000
return sample
tedlium = load_dataset("esb/datasets", "earnings22", split='validation', trust_remote_code=True)
# compute duration to filter
tedlium = tedlium.map(add_duration)
tedlium = tedlium.select(range(512))
# Whisper max supported duration
tedlium… See the full description on the dataset page: https://huggingface.co/datasets/D4nt3/esb-datasets-earnings22-validation-tiny-filtered.instructpix2pix-clip-filtered
Dataset Card for InstructPix2Pix CLIP-filtered
Dataset Summary
The dataset can be used to train models to follow edit instructions. Edit instructions
are available in the edit_prompt. original_image can be used with the edit_prompt and
edited_image denotes the image after applying the edit_prompt on the original_image.
Refer to the GitHub repository to know more about
how this dataset can be used to train a model that can follow instructions.
Supported Tasks… See the full description on the dataset page: https://huggingface.co/datasets/timbrooks/instructpix2pix-clip-filtered.cmmlu_filteredTraditional-Chinese-Common-Crawl-Filtered
Traditional Chinese C4
Dataset Summary
Data obtained from 2013～2025 Common Crawl.
Downloaded and processed using code based on another project attempting to recreate the C4 dataset.
The resultant dataset contains both simplified and traditional Chinese, which could be found here.
It was then filtered using a modified list of simplified Chinese characters to obtain this traditional Chinese dataset.
Unfortunately, I don't have enough funding to run a deduplication across… See the full description on the dataset page: https://huggingface.co/datasets/jed351/Traditional-Chinese-Common-Crawl-Filtered.filtered-wit
Filtered WIT, an Image-Text Dataset.
A reliable Dataset to run Image-Text models.
You can find WIT, Wikipedia Image Text Dataset, here
Data was taken from dalle-mini/wit
Author
Aarush Katta
Data Structure
The data is stored as tars, containing 10,000 samples per tar.
The parquets contain the metadata of each tar, which was crated using this script
Each tar contains a .jpg, .txt, and .json.
The image is stored in .jpg, the caption in .txt. and the metadata in… See the full description on the dataset page: https://huggingface.co/datasets/laion/filtered-wit.google-landmark-v2-chinese-filtered
Google Landmark V2 Chinese Filtered Dataset
This dataset contains landmark images and metadata for training landmark retrieval models, with Chinese translations of landmark names to facilitate Chinese multimodal retrieval tasks.
Dataset Source
This dataset is based on the Google Landmarks V2 dataset from Kaggle. The original data has been filtered and processed to create a high-quality training dataset for landmark retrieval.
Key Features
Filtered… See the full description on the dataset page: https://huggingface.co/datasets/86Cao/google-landmark-v2-chinese-filtered.stackv2_edu_filtered
Stack V2 Edu
Description
We filter the Stack V2 to only include code from openly licensed repositories, based on the license detection performed by the creators of Stack V2. When multiple licenses are detected in a single repository, we ensure that all of the licenses are on the Blue Oak Council certified license list. Per-document license information is available in the license entry of the metadata field of each example. Code for collecting, processing, and preparing… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/stackv2_edu_filtered.OmniThoughtV_Filter_0.5M
Dataset Introduction
OmniThoughtV is a large-scale multimodal long-chain-of-thought dataset distilled from the FineVision dataset using Alibaba Cloud's AI platform (PAI) distillation toolkit, EasyDistill. This dataset establishes a transparent and reproducible data distillation pipeline, enabling efficient construction of multimodal reasoning chains of thought. Fine-tuning smaller models with this dataset effectively endows them with stronger reasoning capabilities and enhances… See the full description on the dataset page: https://huggingface.co/datasets/alibaba-pai/OmniThoughtV_Filter_0.5M.mala-monolingual-filter
MaLA Corpus: Massive Language Adaptation Corpus
This is a cleaned version with some necessary data cleaning.
Dataset Summary
The MaLA Corpus (Massive Language Adaptation) is a comprehensive, multilingual dataset designed to support the continual pre-training of large language models. It covers 939 languages and consists of over 74 billion tokens, making it one of the largest datasets of its kind. With a focus on improving the representation of low-resource… See the full description on the dataset page: https://huggingface.co/datasets/MaLA-LM/mala-monolingual-filter.aops_forum_filteredState-Tahoe-Filtereddroid_filter_lerobot
DROID (filtered) — LeRobot v2.1
A filtered subset of the DROID robot manipulation
dataset, converted to the LeRobot v2.1 format.
Summary
Field
Value
Codebase version
v2.1
Robot
Franka (7-DoF, EEF-delta control)
FPS
15
Episodes
91,070
Frames
24,442,634
Tasks
31,308
Videos
273,210 (3 camera views per episode)
Chunks
92 (chunk size 1000)
Features
Cameras (3× 160×256×3, h264): observation.images.primary (exterior)… See the full description on the dataset page: https://huggingface.co/datasets/periphanes/droid_filter_lerobot.SWE-bench_Lite_filteredlibritts_r_filtered
Dataset Card for Filtered LibriTTS-R
This is a filtered version of LibriTTS-R. It has been filtered based on two sources:
LibriTTS-R paper [1], which lists samples for which speech restoration have failed
LibriTTS-P [2] list of excluded speakers for which multiple speakers have been detected.
LibriTTS-R [1] is a sound quality improved version of the LibriTTS corpus which is a multi-speaker English corpus of approximately
585 hours of read English speech at 24kHz sampling rate… See the full description on the dataset page: https://huggingface.co/datasets/parler-tts/libritts_r_filtered.peS2o_filtered
PeS2o
Description
This dataset is a version of the peS2o dataset restricted to openly licensed articles.
PeS2o is derived from S2ORC, a corpus of openly licensed abstract and full-text papers that have been converted to a structured format using Grobid.
Starting from Grobid’s XML output, peS2o filters papers that are too short, have incorrect metadata, are in languages other than English, and contain OCR errors using a combination of heuristic- and model-based filtering… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/peS2o_filtered.ideal_scenes_filteredpretrain-nllb-filtered
pretrain-nllb-filtered
Filtered parallel corpus from allenai/nllb for cross-lingual embedding pretraining.
Schema
{"query": "string", "pos": ["string", ...]}
query: source language sentence
pos: target language sentence(s)
Configs (51 language pairs)
Config
Parquet Files
nllb_arb_Arab-eng_Latn
20
nllb_ben_Beng-eng_Latn
6
nllb_bul_Cyrl-eng_Latn
15
nllb_cat_Latn-eng_Latn
5
nllb_ces_Latn-eng_Latn
14
nllb_dan_Latn-eng_Latn
12… See the full description on the dataset page: https://huggingface.co/datasets/datalama/pretrain-nllb-filtered.Magpie-Pro-300K-Filtered
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Pro-300K-Filtered.UltraFineWeb-filteredmosaic-starcoder-filtered
Mosaic format for filtered starcoder dataset to train Malaysian LLM
This repository is to store dataset shards using mosaic format.
prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-starcoder.ipynb
using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer
4096 context length.
how-to
git clone,
git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-starcoder-filtered
load it,
from streaming import… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/mosaic-starcoder-filtered.GeneralThought-430K-filteredData from https://huggingface.co/datasets/GeneralReasoning/GeneralThought-430K, removed prompts with non commercial data
common-pile-filtereddclm-stem-filtered