liswei/Taiwan-Text-Excellence-2B

High quality corpus for Taiwanese culture and Traditional Chinese Taiwan Text Excellence (TTE) Contains high quality news and articles in Traditional Chinese. The data processing pipeline is optimized for LLM performance. Is de-duplicated and cleaned using both rule-based and learning-based filters. E.g., urls/emails/html tags/abnormal characters are cleaned, and numbers (full-width or half-width) are normalized. Contains ~2 billion tokens, measured using BPE… See the full description on the dataset page: https://huggingface.co/datasets/liswei/Taiwan-Text-Excellence-2B.