epfl-dlab/llaza-20B

Llaza Mixture 20B This dataset is a 20B-token pretraining subset built for zip2zip language-model pretraining. It is derived from the full Llaza mixture, which is byte-balanced across four top-level domains: Domain Source Target byte ratio General HuggingFaceFW/fineweb-edu, sample-100BT 50% Code bigcode/the-stack-dedup 20% Math HuggingFaceTB/finemath, finemath-3plus 10% Multilingual epfml/FineWeb2-HQ, 20 language subsets 20% The subset was created from… See the full description on the dataset page: https://huggingface.co/datasets/epfl-dlab/llaza-20B.