mkd-chanwoo/normalized-datasets-for-koreanLLM
Normalized Datasets for Korean LLM (Stage 0.5) This is a comprehensive pretraining dataset for Korean LLMs containing ~944M normalized documents from 42 source datasets across English, Korean, Code, and Science domains. Quick Stats Metric Value Total Documents ~944,545,628 Source Datasets 42 Format JSONL (one JSON object per line) Languages English, Korean Domains English, Korean, Code, Science Pipeline Stage Stage 0.5 (after download, before… See the full description on the dataset page: https://huggingface.co/datasets/mkd-chanwoo/normalized-datasets-for-koreanLLM.
Update Quick Stats, Domain Breakdown, Source Field Mappings, Normalization Statistics (2026-06-02)
Upload the_stack_python_part4.jsonl with huggingface_hub
Upload normalized_meta/korean_web.jsonl with huggingface_hub
Upload normalized_meta/korean_law.jsonl with huggingface_hub
Upload normalized_meta/dartdoc.jsonl with huggingface_hub
Upload normalized_meta/cc_korean.jsonl with huggingface_hub
Upload korean_web.jsonl with huggingface_hub
Upload korean_law.jsonl with huggingface_hub
Upload dartdoc.jsonl with huggingface_hub
Upload cc_korean.jsonl with huggingface_hub
Update README.md
Upload fineweb2_korean_part3.jsonl with huggingface_hub
Upload fineweb2_korean_part2.jsonl with huggingface_hub
Upload fineweb2_korean_part1.jsonl with huggingface_hub
Upload s2orc_part3.jsonl with huggingface_hub
Upload s2orc_part2.jsonl with huggingface_hub
Upload s2orc_part1.jsonl with huggingface_hub
Upload the_stack_python_part3.jsonl with huggingface_hub
Upload the_stack_python_part2.jsonl with huggingface_hub
Update dataset card: 38 datasets, 836M docs, remove emojis
Upload normalized_meta/wanjuan_korean.jsonl with huggingface_hub
Upload normalized_meta/the_stack_python.jsonl with huggingface_hub
Upload normalized_meta/s2orc.jsonl with huggingface_hub
Upload normalized_meta/pubmed_abstracts.jsonl with huggingface_hub
Upload normalized_meta/pile_of_law.jsonl with huggingface_hub
Upload normalized_meta/open_med_text.jsonl with huggingface_hub
Upload normalized_meta/mathpile.jsonl with huggingface_hub
Upload normalized_meta/hplt_english_3.jsonl with huggingface_hub
Upload normalized_meta/hplt_english_2.jsonl with huggingface_hub
Upload normalized_meta/hplt_english_1.jsonl with huggingface_hub
Upload normalized_meta/fineweb2_korean.jsonl with huggingface_hub
Upload normalized_meta/cc100_documents_korean.jsonl with huggingface_hub
Upload normalized_meta/c4_korean.jsonl with huggingface_hub
Upload normalized_meta/algebraic_stack.jsonl with huggingface_hub
Upload normalized_meta/aihub_specialized_corpus.jsonl with huggingface_hub
Upload normalized_meta/aihub_korean_corpus_literature.jsonl with huggingface_hub
Upload pubmed_abstracts.jsonl with huggingface_hub
Upload open_med_text.jsonl with huggingface_hub
Upload mathpile.jsonl with huggingface_hub
Upload hplt_english_3.jsonl with huggingface_hub
Upload hplt_english_2.jsonl with huggingface_hub
Upload cc100_documents_korean.jsonl with huggingface_hub
Upload c4_korean.jsonl with huggingface_hub
Upload algebraic_stack.jsonl with huggingface_hub
Upload aihub_specialized_corpus.jsonl with huggingface_hub
Upload aihub_korean_corpus_literature.jsonl with huggingface_hub
Upload the_stack_python_part1.jsonl with huggingface_hub
Delete the_stack_python_part2.jsonl
Delete the_stack_python_part1.jsonl
Upload wanjuan_korean_part5.jsonl with huggingface_hub