datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wiki_cjk_ocrcode-romance-cjk-wiki
Dataset Card for "code-romance-cjk-wiki"
More Information needed
daily-paper-2026-08-15-cjk-skill-router-blind-spots
Script-Blind Retrieval: Quantifying CJK Trigger Blind Spots in BM25-Style Skill Routers
TL;DR — BM25-style skill routers with ASCII-only tokenizers fail completely for CJK-script queries, but a one-line tokenizer extension recovers 95% accuracy — revealing that principled score normalization can paradoxically collapse gated recall from 95% to 19% by rescaling scores below the injection threshold.
ThakiCloud AI Research · 2026-08-15 · 📝 Tech blog (KO)
Problem… See the full description on the dataset page: https://huggingface.co/datasets/thaki-AI/daily-paper-2026-08-15-cjk-skill-router-blind-spots.cjk-tokenizer-from-scratch
CJK Tokenizer From Scratch
Offline research dump for building Chinese / Japanese tokenizers from scratch.
Download tokenizer.zip (≈464 MB), then:
unzip tokenizer.zip
Contents:
papers/ — 41 free arXiv / ACL PDFs (BPE, WordPiece, Unigram, SentencePiece, Chinese SubChar, Japanese MeCab/Sudachi bake-off, …)
github/ — 23 shallow-cloned repos (minbpe, sentencepiece, tokenizers, jieba, mecab, Sudachi, …)
docs/ — Hugging Face NLP course chapter 6 (train BPE / WordPiece / Unigram)… See the full description on the dataset page: https://huggingface.co/datasets/aicrazy124/cjk-tokenizer-from-scratch.huggingface-cjk-metadata
Dataset Card for HuggingFace-CJK-Metadata
Dataset Summary
This dataset provides structured metadata and documentation extracted from the top 700 most downloaded datasets per language on the Hugging Face Hub for Chinese (zh), Japanese (ja), Korean (ko), and English (en, as a reference). The collection includes both high-level metadata (e.g., size, license, task type) and raw dataset card contents, enabling large-scale, cross-linguistic analysis of data curation… See the full description on the dataset page: https://huggingface.co/datasets/Dasool/huggingface-cjk-metadata.cjk_glyphdeepseek-r1-traces-no-cjkBased on cognitivecomputations/dolphin-r1, but with all samples containing CJK stripped out
I used the following RegExp to detect CJK: /[\u4E00-\u9FFF\u3040-\u30FF\uAC00-\uD7AF]/ and ran it on messages, reasoning and answer
claudy-chat-CJK-5kcj_ko_wordsemilia-cjk-xcodec2-archemilia-yodas-cjk-xcodec2cat-dog-imagescjk-test-dstwitter-xiaollei404-2026.01.26-2015845590773764214-cjkRwlBlNKbHx8xr-part1emilia-yodas-cjk-xcodec2-archemilia-cjk-xcodec2-exampleemilia-yodas-cjk-xcodec2-exampleemilia-cjk-xcodec2cjkoree3Uuocr-render-compare-cjk
Japanese and Chinese born-digital Wikipedia pages
400 Japanese and Simplified Chinese pages rendered at 200 dpi from 60 Wikipedia article PDFs, whose exact ground truth is the PDF text layer (page_cer), OCR'd by Qwen3.5-122B-A10B, Qwen2-VL-2B and PP-OCRv5.
Part of a set of datasets released with a benchmark for evaluating document OCR without ground truth by render-and-compare: the OCR output is drawn back into a page image and compared with the original. All datasets:… See the full description on the dataset page: https://huggingface.co/datasets/gt-free-ocr-metrics/ocr-render-compare-cjk.