mkd-chanwoo/normalized-datasets-for-koreanLLM

Normalized Datasets for Korean LLM (Stage 0.5)

This is a comprehensive pretraining dataset for Korean LLMs containing ~944M normalized documents from 42 source datasets across English, Korean, Code, and Science domains.

Quick Stats

Total Documents by Domain

What is Normalization?

This dataset standardizes 42 source datasets with different formats into a unified JSONL schema. Each source uses different field names (e.g., TEXT, plain_text, content), so normalization:

1. 1.Reads each source in its original format
2. 2.Extracts text from its specific field
3. 3.Writes to unified schema with consistent field names
4. 4.Adds metadata (domain, language, doc_id, license)

Does NOT: modify text, filter, deduplicate, re-encode, or translate

Unified Document Schema

{
  "doc_id":             "gutenberg_000000042",
  "source_name":        "gutenberg",
  "domain":             "english",
  "language":           "en",
  "text":               "The full original text of the document...",
  "url":                "https://source-url.com/page",
  "license":            "Public Domain",
  "source_file":        "data/raw/gutenberg/train-00000-of-00001.parquet",
  "source_index":       42,
  "timestamp":          "2026-03-15T08:22:11Z",
  "processing_version": "v2"
}

Source Datasets & Field Mappings

Normalization Statistics (Seen vs Written)

Datasets processed by the Stage 0.5 normalizer (tracked in checkpoint). Datasets not listed below were normalized via legacy path.

⚠️ aihub_korean_corpus_literature: raw JSON 구조 미스매치로 정규화율 매우 낮음. 추후 extraction path 수정 필요.

Key Features

• —Resumable Processing: Checkpoint system allows resuming interrupted normalization
• —High Normalization Rate: Most datasets achieve >99% normalization (minimal parsing failures)
• —Mixed Licenses: Includes public domain, CC, MIT, Apache, and restricted licenses
• —Same Tokenizer: Uses Keural SentencePiece tokenizer (mkd-ai/keural-tokenizer) for token counting

Pipeline Context

Stage 0: Raw Download (~1.5 TB)
    ↓
Stage 0.5: Normalization → ~944M docs (YOU ARE HERE)
    ↓
Stage 1: Filtering → ~704M docs
    ↓
Stage 2: Dedup + Shard → ~512B tokens

License Considerations

Commercial use varies by source:

• —✅ OK: FineWeb, Wikipedia, OpenWebText, C4, arXiv, PubMed (with conditions)
• —⚠️ Caution: CCNews, Falcon RefinedWeb, some medical sources
• —❌ No Commercial: NamuWiki (CC BY-NC-SA), AIHub datasets (research only)
• —🔒 Restricted: StarCoderData, The Stack (BigCode OpenRAIL-M)

Review individual source licenses before commercial use.