astrai_pluto
Pluto-Nano-1.0-Pretrain-v2
ASTRAI Pluto Nano 1.0 — Pretrain Mix (v2)
Curated multilingual pretraining corpus (~50 GB parquet, ~12 B tokens after tokenization) used for ASTRAI Pluto Nano 1.0, a 1 B-total / 50 M-active MoE model with 64 k vocabulary and 5 target languages (EN, PT, ES, ZH, HI).
v2 additions vs v1: OpenThoughts3 (CoT reasoning), openstax textbooks + peS2o (science), and reweighting for better balance. NOTE: factsense (openbmb) was used at training time but is not redistributed here due to its… See the full description on the dataset page: https://huggingface.co/datasets/ASTRAI-labs/Pluto-Nano-1.0-Pretrain-v2.Pluto-Nano-1.0-Pretrain
ASTRAI Pluto Nano 1.0 — Pretrain Mix
Curated multilingual pretraining corpus (~37 GB parquet, ~10 B tokens after tokenization) used for the base pretrain of ASTRAI Pluto Nano 1.0, a 1 B-total / 47 M-active MoE model with 64 k vocabulary and 5 target languages (EN, PT, ES, ZH, HI).
Source mix
Weight
Category
Source
License
Local subdir
30 %
EN Web
HuggingFaceFW/fineweb-edu (sample-350BT)
ODC-BY 1.0
en_fineweb_edu/
10 %
EN Web
HuggingFaceFW/fineweb-edu… See the full description on the dataset page: https://huggingface.co/datasets/ASTRAI-labs/Pluto-Nano-1.0-Pretrain.