Modelpublic

sorryhyun/anima-vocab-pack-cjk

sourceHugging Faceotherupdated 4d agoView on Hugging Face

3likes

Model Card

Anima CJK vocab pack (preview)

<details> <summary>Settings for this image</summary>

anima-base-v1.0, anima_cjk_vocab_pack_preview, seed 0, 30 steps, CFG 4, 768×1344, no LoRA.

prompt:   sensitive, 1girl, hoshino ai, oshi no ko, @akipeko, looking at viewer, solo, photo background, indoors, skirt, v, speech bubble, japanese text. She is at the bar. Japanese text reads as "はい".
negative: blurry, score_1, monochrome, jpeg artifact, thick border

</details>

ComfyUI: load the pack with the AnimaVocabPackLoader node (MODEL, CLIP, vocab_pack) → (MODEL, CLIP) from ComfyUI-Anima_lora-Adapter. Put the .safetensors and .json of a pack side by side and select the .safetensors. A regular LoRA loader cannot load it.

The pack is anima_cjk_vocab_pack_preview.safetensors plus its .json sidecar. 503 of its rows are trained to draw quoted Japanese text.

Why extend T5?

Anima can already draw many CJK glyphs. That ability lives in the DiT, but no prompt can reach it: the T5 tokenizer has no ids for Japanese, so a quoted "はい" never becomes a usable address.

This pack adds new T5-side token rows instead of changing the model. The rows are used only when a prompt contains CJK characters or symbols T5 cannot spell. Everything else tokenizes exactly as before, so English prompts are bit-identical with or without the pack.

The DiT, the LLM adapter and both text encoders stay untouched. Anything that works on base Anima still works, including LoRA training. For example, you can train a Japanese illustration LoRA with OCR text in the captions and use it together with the pack.

How it was trained

The new rows are the only trainable parameters. They are trained with the ordinary flow-matching loss, so the gradient flows back through the frozen DiT and the frozen adapter and lands only on the rows. The training images are scene composites with Japanese text in speech bubbles, and the loss is weighted up inside the text box.

Anima's text side, with every weight frozen. Qwen3 reads the raw caption and supplies the adapter's keys and values. The T5 id stream is the query: English keeps its stock ids, and each Japanese piece is routed to a pack row with id ≥ 32128. English captions never touch a pack row.

Curious how it works? See the write-up: Waking Anima up to read Japanese.

Usage in detail

This is a vocab pack, not a LoRA. It holds 69,558 extra text-embedding rows (ext_embed [69558, 1024], fp32) appended after the base table, so their ids start at 32128. The .json sidecar carries the tokenization and segmentation maps and the routing rule. Both files must sit together with the same stem.

A pack changes exactly two things in an Anima stack:

1.The T5-side id stream. CJK spans are re-tokenized with the Qwen3 tokenizer and mapped to pack rows.
2.The adapter's id table (llm_adapter.embed). The pack rows are appended after the 32,128 stock rows.

The ComfyUI node and animalora do both steps for you. diffusers has no pack support yet, so there both steps are done by hand. The routing logic is one file, [`library/anima/extvocab.py`](https://github.com/sorryhyun/animalora/blob/main/library/anima/extvocab.py), which needs only the standard library and torch. Install anima_lora or copy that file.

Runnable versions of both snippets are `examples/09_cjk_vocab_pack.py` (animalora engine) and [`examples/10cjkvocabpackdiffusers.py`](https://github.com/sorryhyun/animalora/blob/main/examples/10cjkvocabpackdiffusers.py) (diffusers). Each has a --dry_run flag that prints the routed id stream without loading model weights.

diffusers (≥ 0.39, `ModularPipeline`)

python

import torch
from huggingface_hub import hf_hub_download
from diffusers.modular_pipelines.anima import AnimaAutoBlocks
from diffusers.modular_pipelines.anima.encoders import AnimaTextEncoderStep
from library.anima.ext_vocab import HybridT5Encoder, load_ext_assets  # anima_lora repo

REPO, PACK = "sorryhyun/anima-vocab-pack-cjk", "anima_cjk_vocab_pack_preview"
prefix = hf_hub_download(REPO, f"{PACK}.safetensors")[:-len(".safetensors")]
hf_hub_download(REPO, f"{PACK}.json")
table, mapping = load_ext_assets(prefix)

# 1. text_encoder block: T5 ids through the pack (stock ids for prompts with no routed character)
class PackTextEncoderStep(AnimaTextEncoderStep):
    _enc = None
    @classmethod
    def _get_t5_prompt_ids(cls, components, prompt, max_sequence_length, device):
        if cls._enc is None:  # the pipeline's own tokenizers: same vocabularies the pack was built on
            cls._enc = HybridT5Encoder.from_mapping(components.t5_tokenizer, components.tokenizer, mapping)
        prompt = [prompt] if isinstance(prompt, str) else prompt
        rows = []
        for t in prompt:
            if cls._enc.routes(t):
                ids, mask = cls._enc.encode(t, max_sequence_length)
                rows.append(ids[: sum(mask)])
            else:
                rows.append(components.t5_tokenizer(t, max_length=max_sequence_length, truncation=True)["input_ids"])
        n = max(map(len, rows)); pad = components.t5_tokenizer.pad_token_id
        ids = torch.tensor([r + [pad] * (n - len(r)) for r in rows])
        mask = torch.tensor([[1] * len(r) + [0] * (n - len(r)) for r in rows])
        return ids.to(device), mask.to(device)

blocks = AnimaAutoBlocks()
blocks.sub_blocks["text_encoder"] = PackTextEncoderStep()
pipe = blocks.init_pipeline("circlestone-labs/Anima-Base-v1.0-Diffusers")
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")

# 2. widen the conditioner's id table once (dtype/device follow it)
emb = pipe.text_conditioner.embed
pipe.text_conditioner.embed = torch.nn.Embedding.from_pretrained(
    torch.cat([emb.weight.data, table.to(emb.weight.dtype).to(emb.weight.device)])
)

image = pipe(prompt="1girl, 猫耳, 銀髪, セーラー服, 笑顔, 教室", num_inference_steps=30).images[0]

LoRAs loaded with pipe.load_lora_weights(...) compose with the pack, because they touch different parameters.

anima_lora

In anima_lora, a pack is one field on the request. The engine installs the routing tokenizer and appends the rows to llm_adapter.embed for you.

python

import torch
from huggingface_hub import hf_hub_download
from anima_lora import GenerationRequest, default_checkpoints, generate, get_generation_settings, load_vae, save_output

REPO, PACK = "sorryhyun/anima-vocab-pack-cjk", "anima_cjk_vocab_pack_preview"
pack = hf_hub_download(REPO, f"{PACK}.safetensors")
hf_hub_download(REPO, f"{PACK}.json")

ckpt = default_checkpoints()
args = GenerationRequest(dit=ckpt.dit, vae=ckpt.vae, text_encoder=ckpt.text_encoder,
                         vocab_pack=pack,  # a .safetensors path or its prefix; no_vocab_pack=True turns it off
                         prompt="1girl, 猫耳, 銀髪, セーラー服, 笑顔, 教室", save_path="out.png").to_args()
args.device = device = torch.device("cuda")
latent = generate(args, get_generation_settings(args))

vae = load_vae(args.vae, device="cpu", dtype=torch.bfloat16, eval=True)
save_output(args, vae, latent, device)

Training uses the same key: train.py --vocab_pack <prefix>. The LoRA is stamped with the pack digest (ss_ext_pack_sha), so loaders can warn when it is used with a different pack.

Limitations

—Rendering is at an early stage. As of 2026-09-17, only a small set of Japanese characters has been trained to render. Single kana and kanji and very short words like はい render some of the time. Longer words and sentences mostly do not. Results depend on the seed, so generate a few.
—The pack cannot exceed the base model. It only gives prompts a way to reach glyphs Anima already knows. It cannot draw text more accurately or more cleanly than Anima itself can.
—Japanese prompts may drift. The rendering rows were trained inside English caption frames such as Japanese text reads as "…". Prompts that use the same Japanese characters in other ways, like Japanese tags, may behave differently than they did before rendering training. English prompts are unaffected.
—Only `anima-base-v1.0` was used for training. Other Anima checkpoints are untested.

Roadmap

—Reliable rendering of words and sentences
—Rendering for all 2,136 jōyō kanji
—Rendering for Korean and Chinese