Datasetpublic

ZhejiangLab/CPT_Data_Pool

CPT Data Pool This repository hosts a large-scale CPT corpus designed for continual pre-training of domain-specific large language models. It serves as a data component of the 👉F2D-LLM framework, an end-to-end pipeline for domain-specific LLM training. For detailed data processing, scoring methods, and sampling strategies, please refer to the official 👉GitHub repo. Overall, the dataset contains approximately 271B tokens and is stored as jsonl files with a unified schema… See the full description on the dataset page: https://huggingface.co/datasets/ZhejiangLab/CPT_Data_Pool.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

1likes4.7kdownloads

Dataset Card

CPT Data Pool

This repository hosts a large-scale CPT corpus designed for continual pre-training of domain-specific large language models. It serves as a data component of the 👉F2D-LLM framework, an end-to-end pipeline for domain-specific LLM training.

For detailed data processing, scoring methods, and sampling strategies, please refer to the official 👉GitHub repo.

Overall, the dataset contains approximately 271B tokens and is stored as jsonl files with a unified schema, covering mathematical reasoning, programming code, academic knowledge and general knowledge. All data sources are annotated with quality score tags and the sampling ratios below have been experimentally validated as an effective mix.

Category	Size (tokens)	Category Ratio	Source	Source Ratio	Source License	Score Tag
Mathematical reasoning	34B	35%	Dolma 1.7-AlgebraicStack	12%	ODC‑BY	`math-score`
			Dolma 1.7-OpenWebMath	11%	ODC‑BY	`math-score`
			MegaMath-Web-Pro	12%	ODC‑BY	`math-score`
Programming code	100B	20%	StarCoder	20%	Original repository permissive licenses	`code-score`
Academic knowledge	77B	15%	Dolma 1.7-RedPajama-arXiv	10%	ODC‑BY	`edu-score`
			Pes2o	5%	ODC‑BY	`edu-score`
General knowledge	60B	30%	Dolma 1.7-CC News	4%	ODC‑BY	`edu-score`
			Dolma 1.7-Wiki	6%	ODC‑BY	`edu-score`
			Dolma 1.7-Books	4%	ODC‑BY	`edu-score`
			Dolma 1.7-MegaWika	5%	ODC‑BY	`edu-score`
			Dolma 1.7-Reddit	1%	ODC‑BY	`edu-score`
			Dolma 1.7-Flan	10%	ODC‑BY	`edu-score`
Total	271B	100%	-	100%	-	-

</div>

Important: The CPT Data Pool is aggregated from multiple publicly available sources, each of which is subject to its own license. While the scripts and preprocessing workflow are licensed under the Apache License 2.0, redistribution of the dataset must comply with the original data licenses. Users are responsible for adhering to the terms of each individual data source when using or redistributing the source data.