climbmix
Datasets
All datasets matching “climbmix”climbmix-400b-shuffleNemotron-ClimbMix
ClimbMix Dataset
🚀 Creating the highest-quality pre-training datasets for LLMs 🌟
📄 PAPER
🤗 CLIMBLAB
🤗 CLIMBMIX
🏠 HOMEPAGE
Figure 1: Continuously training a 1B model yields a 2.0% improvement over Llama-3.2-1B, demonstrating a more efficient scaling trend compared to prior models.
Figure 2: Pre-training a 1B model from scratch on ClimbMix shows better scaling effects than training on other datasets.… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix.ClimbMix
ClimbMix
About
🧗 A more convenient ClimbMix (https://arxiv.org/abs/2504.13161)
Description
Unfortunately, the original ClimbMix (https://huggingface.co/datasets/nvidia/ClimbMix) has four main inconveniences:
It is in GPT2 tokens, meaning you have to detokenize it to inspect it or use it with another tokenizer.
It contains all of the 20 clusters in order together (in the same "subset"), so you have to load the whole dataset in memory (~1TB) and shuffle it… See the full description on the dataset page: https://huggingface.co/datasets/gvlassis/ClimbMix.nanochat-climbmix-arithmetic-base10
nanochat ClimbMix + Base-10 Arithmetic
This dataset contains the first 170 shuffled ClimbMix training shards
used by nanochat's speedrun. The deterministic base-10 arithmetic corpus is
mixed into shards 00000..00149; the final
20 train shards are unchanged web-only padding.
The original validation shard (shard_06542.parquet) is also
copied unchanged.
Arithmetic corpus
Family
Examples
a + b = c (all ordered pairs 0..2000, two exposures)
8,008,002
a + b… See the full description on the dataset page: https://huggingface.co/datasets/Yujivus/nanochat-climbmix-arithmetic-base10.ClimbMixClimbMix is a high-quality pre-training corpus released by NVIDIA. Here is the description:
ClimbMix is a compact yet powerful 400-billion-token dataset designed for efficient pre-training that delivers superior performance under an equal token budget. It was introduced in this paper.
We proposed a new algorithm to filter and mix the dataset. First, we grouped the data into 1,000 groups based on topic information. Then we applied two classifiers: one to detect advertisements and another to… See the full description on the dataset page: https://huggingface.co/datasets/OptimalScale/ClimbMix.ClimbMix10M
climbmix10M
About
Subsampled version of gvlassis/ClimbMix containing 10,000,000 samples while preserving the original 20-cluster ratio distribution.
Description
This dataset is created by stream-sampling from gvlassis/ClimbMix without downloading the full 553M rows. The sampling preserves the exact ratio distribution across all 20 clusters using the largest-remainder method.
Per-cluster quotas:
cluster_id
topics
documents
ratio
1… See the full description on the dataset page: https://huggingface.co/datasets/aimlresearch2023/ClimbMix10M.