datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SWE-smith
SWE-smith Dataset
Code
•
Paper
•
Site
[12/14/2025] NOTE: We will no longer actively update this dataset.
While this dataset is still functional and usable, we recommend you use the `SWE-bench/SWE-smith-[lang]` datasets.
For better maintainability and ease-of-use, we are maintaining language-specific datasets in lieu of this mono-repo.
The SWE-smith Dataset is a training dataset of 50137 task instances from 128 GitHub repositories, collected using the SWE-smith toolkit.… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-smith.SWE-smith-py
SWE-smith Dataset
Code
•
Paper
•
Site
As of 12/14/2025, SWE-smith: Python contains 50908 task instances from 131 GitHub repositories
The SWE-smith Dataset is the largest open source dataset for training software engineering agents.
All SWE-smith task instances come with an executable environment.
To learn more about how to use this dataset to train Language Models for Software Engineering, please refer to the documentation.
SWE-smith-pythonSWE-smith-javaSWE-smith2SWE-smith-trajectories
SWE-smith Trajectories
Code
•
Paper
•
Site
This dataset contains the 5017 trajectories we fine-tuned Qwen 2.5 Coder Instruct on, leading to
SWE-agent-LM-32B, a coding LM agent that
achieve 40.2% on SWE-bench Verified (no verifiers or multiple rollouts, just 1 attempt per instance).
Trajectories were generated by running SWE-agent + Claude 3.7 Sonnet on task instances from
the SWE-smith dataset.
SWE-smith-jsMulti-SWE-smith-tasksSWE-smith-tsminty-astro-ph
MINT-1T ArXiv Astro-ph
An astronomy-focused subset of mlfoundations/MINT-1T-ArXiv, filtered to include only papers from the astro-ph arXiv category (including cross-listed papers).
Overview
Papers
~845k
Total size
~804 GB
Format
WebDataset tar shards
Shards
287 (astro-ph-00000.tar to astro-ph-00286.tar)
Shard size
~3 GB each
Source
MINT-1T (Awadalla et al., 2024)
Data Format
Each tar shard contains paired files per paper:… See the full description on the dataset page: https://huggingface.co/datasets/Smith42/minty-astro-ph.SWE-smith-cppSWE-smith-go
SWE-smith Dataset
Code
•
Paper
•
Site
As of 12/14/202, SWE-smith: Golang contains 8212 task instances from 87 GitHub repositories
The SWE-smith Dataset is the largest open source dataset for training software engineering agents.
All SWE-smith task instances come with an executable environment.
To learn more about how to use this dataset to train Language Models for Software Engineering, please refer to the documentation.
danbooru-v4-tfrecordDanbooru TFRecords to train classifiers and other stuff with my codebases.
TFRecord serialization/deserialization code:
NUM_CLASSES = 12822
# Function to convert value to bytes_list
def _bytes_feature(value):
if isinstance(value, type(tf.constant(0))):
value = value.numpy()
elif isinstance(value, str):
value = value.encode()
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
# Function to convert bool/enum/int/uint to int64_listdef… See the full description on the dataset page: https://huggingface.co/datasets/SmilingWolf/danbooru-v4-tfrecord.galaxies
Galaxies for training astroPT
Here we have ~8.5 million galaxy cutouts from the DESI legacy survey DR8.
The cut outs are 512x512 pixel jpg images centred on the galaxy source.
I've split away 1% of the images into a test set, and 1% into a validation set.
The remaining 98% of the images comprise the training set.
Metadata (galaxy properties)
There is also accompanying metadata!
To combine the metadata with the galaxy images you can do (for example):
from datasets import… See the full description on the dataset page: https://huggingface.co/datasets/Smith42/galaxies.SWE-smith-mini_swe_agent_plus-trajectories-66k
Dataset: SWE-smith-mini_swe_agent_plus-trajectories-66k
A corpus of ~66k issue-solving trajectories collected with mini-swe-agent-plus on issues derived from SWE-smith. Each trajectory records the agent’s end-to-end process.
We training the Qwen3-8B model on different sizes of the training data. The results are shown in the figure, it could be observed that the solve rate on SWE-bench Verified improves approximately linearly with the logarithm of the data scale (1k → 66k… See the full description on the dataset page: https://huggingface.co/datasets/Kwai-Klear/SWE-smith-mini_swe_agent_plus-trajectories-66k.smiles-transformers
smiles-transformers dataset
TODO: Add references to the datasets we curated
dataset features
name: text
Molecule SMILES : string
name: formula
Molecular formula : string
name: NumHDonors
Number of hidrogen bond donors : int
name: NumHAcceptors
Number of hidrogen bond acceptors : int
name: MolLogP
Wildman-Crippen LogP : float
name: NumHeteroatoms
Number of hetero atoms: int
name: RingCount
Number of rings : int
name: NumRotatableBonds
Number of rotable… See the full description on the dataset page: https://huggingface.co/datasets/maykcaldas/smiles-transformers.SWE-smith-oracleThis is a version of SWE-bench/SWE-smith filtered for non-empty problem_statement and formatted into the oracle setting of SWE-bench where the files edited by the patch are displayed to the agent. This problem presentation is made available in a text column, following the format of princeton-nlp/SWE-bench_Lite_oracle.
SWE-smith-rsrepo2rlenv-swe-smith
Repo2RLEnv SWE-smith
Apply owned procedural mutations to real repository functions, require an executable test contrast, and author a natural issue from bounded evidence.
Contains 100 Harbor tasks generated with the owned
swe_smith recipe in Repo2RLEnv.
Browse the complete task bundles in Harbor Visualiser or
open the task folders. Each folder is a runnable Harbor task:
tasks/<task_id>/
├── task.toml # Harbor configuration and provenance
├── instruction.md… See the full description on the dataset page: https://huggingface.co/datasets/FineEnvs/repo2rlenv-swe-smith.e621-v1-tfrecordE621 TFRecords to train classifiers and other stuff with my codebases.
TFRecord serialization/deserialization code:
NUM_CLASSES = 8783
# Function to convert value to bytes_list
def _bytes_feature(value):
if isinstance(value, type(tf.constant(0))):
value = value.numpy()
elif isinstance(value, str):
value = value.encode()
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
# Function to convert bool/enum/int/uint to int64_listdef… See the full description on the dataset page: https://huggingface.co/datasets/SmilingWolf/e621-v1-tfrecord.smithsonian_butterflies_subsetThis a subset of "ceyda/smithsonian_butterflies" dataset with additional processing done to train the "ceyda/butterfly_gan" model.
The preprocessing includes:
Adding "sim_score" to images with CLIP model using "pretty butterfly","one butterfly","butterfly with open wings","colorful butterfly"
Removing butterflies with the same name(species)
Limiting only to the top 1000 images
Removing the background (doing another sim_scoring after bg removal did visually worse so didn't do it)
Detecting… See the full description on the dataset page: https://huggingface.co/datasets/huggan/smithsonian_butterflies_subset.SWE-smith-phpSWE-smith-filteredFiltered dataset from SWE-bench/SWE-smith that the problem_statement is not empty for quick loading.
After removing the rows with empty problem_statement, it comes to 52945 -> 10924 rows (removed 42021 empty problem_statements)
SWE-smith-trajectories-harborimport datasets
def map_to_harbor(example):
# in image name, change jyangballin/ for swesmith/
example['image_name'] = example['image_name'].replace('jyangballin/', 'swesmith/')
return example
smith = datasets.load_dataset('SWE-bench/SWE-smith', split='train')
smith = smith.map(map_to_harbor)
smith = datasets.DatasetDict({'train': smith})
smith.push_to_hub('ricdomolm/SWE-smith-trajectories-harbor')
SWE-smith-js-4259-filteredAISHELL-3
AISHELL-3
Dataset Description
AISHELL-3 is a large-scale and high-fidelity multi-speaker Mandarin speech corpus published by Beijing Shell Shell Technology Co., Ltd.
The dataset is designed for training multi-speaker Text-to-Speech (TTS) systems. It contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Mandarin Chinese speakers, with a total of 88,035 utterances.
Auxiliary speaker attributes, including gender, age group, and native accents, are… See the full description on the dataset page: https://huggingface.co/datasets/SMIIP-lab/AISHELL-3.SWE-smith-filteredFiltered SWE-smith instances that have problem statements as of May 23rd, 2025. Also maximally compressed with categorical variables for low cardinality values.
SWE-smith-jsSWE-smith-java-6704-filtered-for-problem-statementsswe_smith_js_5902_filtered