Datasetpublic

KaraKaraWitch/pixiv-dic-auto-translated

KaraKaraWitch/pixiv-dic-auto-translated Unfiltered Japanese→English machine translation of the Pixiv Encyclopedia. Dataset Summary ~150k Pixiv Encyclopedia head-words and article snippets, raw-translated into English. Translations are public-domain. No blacklist, no bowdlerisation—if the JP page existed, it’s in here. Languages Code Language Role ja Japanese original en English machine target The language: list only contains ja… See the full description on the dataset page: https://huggingface.co/datasets/KaraKaraWitch/pixiv-dic-auto-translated.

sourceHugging Faceunlicenseupdated 9mo agoView on Hugging Face

1likes101downloads

Dataset Card

KaraKaraWitch/pixiv-dic-auto-translated

Unfiltered Japanese→English machine translation of the Pixiv Encyclopedia.

Dataset Summary

~150k Pixiv Encyclopedia head-words and article snippets, raw-translated into English. Translations are public-domain. No blacklist, no bowdlerisation—if the JP page existed, it’s in here.

Languages

Code	Language	Role
ja	Japanese	original
en	English	machine target

The language: list only contains ja and en because every record is JP→EN. The handful of KR/ZH characters that occasionally appear in orig are stray loan-terms already present in the Japanese wiki; they are still translated to English, not Korean/Chinese.

Data Fields

Field	Type	Description
`orig`	string	Japanese head-word (or occasional CJK loan)
`rom`	string	Furigana / reading hint
`breadcrumbs`	string	Pixiv category path (JP)
`eng`	string	English head-word (manual fallback or romanisation)
`summary`	string	First paragraph of the article, translated
`classification`	enum	Danbooru-style tag type (see below)
`normalized_ja`	string	Cleaned JP head-word for dedup

Classification Labels

python

class TagType(Enum):
    Artist     = "Artist"
    Copyright  = "Copyright"
    Character  = "Character"
    General    = "General"
    Meta       = "Meta"
    Event      = "Event"
    Emotion    = "Emotion"
    Object     = "Object"
    Setting    = "Setting"
    Action     = "Action"
    Body       = "Body"
    Clothing   = "Clothing"
    Style      = "Style"
    Parody     = "Parody"
    Redirect   = "Redirect"
    Ambiguous  = "Ambiguous"
    Other      = "Other"

(Label set borrowed from Danbooru but applied to encyclopedia pages, not image tags.)

Generation Setup

—Model: KaraKaraWitch/GoldDiamondGold-L33-70b
—Prompt template: see prompt.md
—Decoding: structured generation (Pydantic/Json Schema) to guarantee enum values & JSONL format.
—Post-processing: none—output is raw 70-B shot minus whitespace fixes.

Use at Your Own Risk

Content is not curated. Expect NSFW lemmas, slurs, obsolete memes, and Pixiv in-jokes. If you need a family-friendly slice, filter on classification and do your own second-pass cleaning.

Citation

bibtex

@misc{pixivdictrans2025,
  title={{Pixiv Dictionary: Translated}},
  author={KaraKaraWitch},
  howpublished={\url{https://huggingface.co/datasets/KaraKaraWitch/pixiv-dic-auto-translated}},
  year={2025},
  note={Unlicense}
}