datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Nemotron-ClimbMix
ClimbMix Dataset
🚀 Creating the highest-quality pre-training datasets for LLMs 🌟
📄 PAPER
🤗 CLIMBLAB
🤗 CLIMBMIX
🏠 HOMEPAGE
Figure 1: Continuously training a 1B model yields a 2.0% improvement over Llama-3.2-1B, demonstrating a more efficient scaling trend compared to prior models.
Figure 2: Pre-training a 1B model from scratch on ClimbMix shows better scaling effects than training on other datasets.… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix.ClimbMixClimbMix is a high-quality pre-training corpus released by NVIDIA. Here is the description:
ClimbMix is a compact yet powerful 400-billion-token dataset designed for efficient pre-training that delivers superior performance under an equal token budget. It was introduced in this paper.
We proposed a new algorithm to filter and mix the dataset. First, we grouped the data into 1,000 groups based on topic information. Then we applied two classifiers: one to detect advertisements and another to… See the full description on the dataset page: https://huggingface.co/datasets/OptimalScale/ClimbMix.nanochat-climbmix-hq
Nanochat climbmix dataset filtered
This repository contains a filtered version of the climbmix dataset for efficient use with Andrej Karpathy’s Nanochat project.
nanochat-climbmix-annotated
Summary
A 200 shards subset of karpathy/climbmix-400b-shuffle dataset (Nvidia ClimbMix) of web documents with added pre-computed embeddings and classified topics and formats.
Parquet files keep the nanochat compatible format (row groups, 'text' column), so this can be used as a drop-in replacement of the Karpathy's mix in the nanochat project, where the additional metadata can be used in the code.
Dataset Structure
Size: 200 parquet shards (~86K rows each, ~16.9M… See the full description on the dataset page: https://huggingface.co/datasets/ddudek/nanochat-climbmix-annotated.ClimbMix-SampleClimbMix_part_1climbmix-30b-juniper
Climbmix-30b-Juniper
A 30B-token pre-tokenized slice of NVIDIA ClimbMix
(via the shuffled karpathy/climbmix-400b-shuffle),
prepared as pre-training data for the Juniper model series.
Dataset Details
Size: 30 billion tokens
Format: ArrayRecord
Tokenizer: o200k_harmony — tiktoken o200k_base plus the gpt-oss special tokens (vocab 201,088)
Shards: 100 ArrayRecord files (group_size:1), exactly 300M tokens each
No BOS token; documents are delimited by EOS only
ClimbMix-sample
Unofficial NVIDIA Nemotron-ClimbMix (Subsampled)
This dataset is a curated, subsampled version of OptimalScale/ClimbMix, which itself is a detokenized version of NVIDIA's official pretraining dataset, nvidia/Nemotron-ClimbMix.
It is designed for researchers and developers looking for a smaller, well-shuffled slice of the Nemotron pretraining data for quick experimentation, testing, or ablation studies.
Processing Method
To create this streamlined version, the… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/ClimbMix-sample.ClimbMix_part_0stride-preproc-climbmix
STRIDE: Preprocessed ClimbMix
Tokenized ClimbMix sequences used for STRIDE's pretraining-data attribution experiments. There is one training pool per released nanochat depth (d12, d16, d20, d24) and one held-out test set shared across depths. These are the corpora the released nanochat operators attribute over, so STRIDE's column indices line up with the lines of these files.
Files
File
Sequences
Size
Contents
climbmix_train_d12.jsonl
1,317,003
3.8 GB
training… See the full description on the dataset page: https://huggingface.co/datasets/CausalNLP/stride-preproc-climbmix.kimi_climbmix_30kClimbMix-split
ClimbMix Split
climbmix-split reorganizes the detokenized NVIDIA ClimbMix source corpus into
source-oriented splits. ClimbMix is described as being built from Nemotron-CC
and SmolLM-Corpus. Because SmolLM-Corpus is the smaller and directly
identifiable component, we used exact normalized-text matching against
SmolLM-Corpus to recover the SmolLM-derived portions. The remaining rows are
provided as the residual nemotron-cc split.
The data rows are unchanged from the detokenized… See the full description on the dataset page: https://huggingface.co/datasets/AIML-TUDA/ClimbMix-split.