CoolFace
Datasetpublic

jakequist/kanjiland-silver-annotations

Kanjiland — Silver Annotations 9,388 Japanese sentences annotated in the full Kanjiland reading-comprehension format: morpheme segmentation, furigana (ruby on kanji runs), per-token contextual glosses, word groupings, an English sentence translation, and grammar-pattern labels from a closed 120-rule inventory. Part of Kanjiland, a from-scratch Japanese reading-comprehension engine. Format Each line of silver_annotations.jsonl is: {"ja": "<Japanese sentence>"… See the full description on the dataset page: https://huggingface.co/datasets/jakequist/kanjiland-silver-annotations.

sourceHugging Facecc-by-sa-3.0updated 2mo agoView on Hugging Face
0likes18downloads
Dataset Card

Kanjiland — Silver Annotations

9,388 Japanese sentences annotated in the full Kanjiland reading-comprehension format: morpheme segmentation, furigana (ruby on kanji runs), per-token contextual glosses, word groupings, an English sentence translation, and grammar-pattern labels from a closed 120-rule inventory.

Part of **Kanjiland**, a from-scratch Japanese reading-comprehension engine.

Format

Each line of silver_annotations.jsonl is:

json
{"ja": "<Japanese sentence>", "wire": "<tagged annotation string>"}

The wire string uses Unicode Private-Use-Area separators (⟨T⟩ token, ⟨W⟩ word, ⟨S⟩ sentence, ⟨G⟩ grammar). The format is specified in `docs/FORMAT_SPEC.md`; the grammar rule inventory is `docs/GRAMMAR_RULES.md`.

How it was made (hybrid supervision)

  • —Source sentences: KFTT (Kyoto Free Translation Task), formal/Wikipedia domain.
  • —Deterministic layer: MeCab + UniDic (offline) — segmentation, kanji-run ruby, POS, lemma.
  • —Judgment layer: an OpenAI teacher model — contextual glosses, translation, grammar labels.
  • —Gate: every annotation passes a strict format linter (93.9% of 10,000 sentences passed).

License & attribution

CC-BY-SA 3.0. Derived from KFTT (© Graham Neubig, CC-BY-SA 3.0, http://www.phontron.com/kftt/) — attribute KFTT and this project, and share derivatives alike. English glosses/translations are model-generated (OpenAI); deterministic labels use MeCab (BSD) + UniDic. See the repo NOTICE.md.

Limitations

  • —Domain: formal/Wikipedia (KFTT) — not conversational or web Japanese.
  • —Silver, not gold: teacher-generated; the linter catches structural errors but not every semantic mislabel. Spot-audited, not exhaustively verified.