mkd-minju/LLMDataProcessing

LLM Data Processing Real, end-to-end LLM data pipeline output: raw Common Crawl WARC records, progressively filtered/cleaned into a pretraining corpus, plus a derived SFT (instruction-tuning) set and a DPO (preference) set. Every file here is the actual output of a script run — no synthetic placeholders — against a real Common Crawl batch (CC-MAIN-2026-25, discovered dynamically at run time, not hardcoded). Source code, full run logs, and the research/decision notes behind every… See the full description on the dataset page: https://huggingface.co/datasets/mkd-minju/LLMDataProcessing.