Datasetpublic

mlfoundations/dclm-baseline-1.0

DCLM-baseline DCLM-baseline is a 4T token / 3B document pretraining dataset that achieves strong performance on language model benchmarks. Below are comparisions of model trained on DCLM-baseline with other models in the 7B regime. Model Params Tokens Open dataset? CORE MMLU EXTENDED Open weights, closed datasets Llama2 7B 2T ✗ 49.2 45.8 34.1 DeepSeek 7B 2T ✗ 50.7 48.5 35.3 Mistral-0.3 7B ? ✗ 57.0 62.7 45.1 QWEN-2 7B ? ✗ 57.5 71.9 50.5 Llama3 8B 15T ✗… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0.

sourceHugging Facecc-by-4.0updated 2y agoView on Hugging Face

314likes633kdownloads

50 commits on main

a3b142c2y ago

Define features explicitly in the dataset card (#11)

achal-tri, albertvillanova

1fbd7c42y ago

Update README.md to add Amber and Crystal from llm360 (#10)

achal-tri, Tianhua

7404a6c2y ago

Fix file extension: .zstd -> .zst (#7)

vaishaal, lhoestq

5b34ef82y ago

Update README.md

gsmyrnis

c312c212y ago

Upload global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed.jsonl.zstd with huggingface_hub

jeffreywpli

41e09cc2y ago

Delete global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed.jsonl.zstd

jeffreywpli

5b392d42y ago

Upload global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed.jsonl.zstd with huggingface_hub

jeffreywpli

ba56baa2y ago

Delete global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed.jsonl.zstd (#3)

jeffreywpli

c30a25c2y ago

Upload global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed.jsonl.zstd with huggingface_hub

jeffreywpli

9b400492y ago

Add missing file

jeffreywpli

04661d52y ago

Rename global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed_old.jsonl.zstd to global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed.jsonl.zstd

jeffreywpli

ccd38332y ago

Rename global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed.jsonl.zstd to global-shard_10_of_10/local-shard_5_of_10/shard_00000031_processed_old.jsonl.zstd

jeffreywpli

82e3b8d2y ago

Update README.md

vaishaal

8030d982y ago

Update README.md

vaishaal

514821e2y ago