hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking

CantoneseLLM-v2.0-8B-Thinking

An 8B dense model based on Qwen3-8B that reasons and answers in Hong Kong Cantonese.

It is the final checkpoint of a five-stage pipeline: continual pre-training → chat-vector merge → supervised fine-tuning → DPO → two-stage RLVR. The full development history, including the stages that failed and why, is documented in the technical report.

📄 Paper: CantoneseLLM v2: Reasoning in a Low-Resource Language (arXiv:2609.06970)

🧪 Evaluation Benchmark: hon9kon9ize/hkeval2025

Reasoning in Cantonese

Open models prompted in Cantonese answer in Cantonese but reason in another language. Here is the same translated GSM8K probe, under a system prompt that explicitly asks for Hong Kong Cantonese.

Chat-vector merged checkpoint — reasons entirely in Simplified Chinese, despite the system prompt and despite continual pre-training on Traditional Chinese and Cantonese:

嗯，用户问的是Janet每天赚多少钱。首先，我需要理清楚整个流程。题目说Janet的鸭子每天生16只蛋。 (Well, what the user is asking is how much Janet earns each day. First, I need to work through the whole process. The problem says Janet's ducks lay 16 eggs a day.)

This model — reasons in Cantonese end to end, with the Cantonese classifier 隻 zek3 in place of 只 and colloquial forms such as 剩返 zing6 faan1:

呢個係一個數學應用題，我需要先理解題目入面嘅數據同埋計算邏輯。Janet 隻鴨每日生 16 隻蛋。首先，佢自己食早餐用咗 3 隻，然後用 4 隻嚟整鬆餅。… 剩返嘅蛋就係 16 − 7 = 9 隻。 (This is a mathematics word problem, and I need to understand the figures and the logic of the calculation first. Janet's ducks lay 16 eggs a day. First, she uses 3 for her own breakfast, then 4 to make muffins. … The eggs left over are 16 − 7 = 9.)

Both reach the correct answer, 18 dollars. What changes is where the reasoning happens. The 只/隻 substitution is not cosmetic: 只 can only be read zi2 in Hong Kong Cantonese, which is a different word from the classifier the sentence needs.

The mechanism is a multiplicative language and script term in the RLVR reward — task_reward × language_multiplier × format_factor — so a wrong-language trace cannot buy its way back by being correct.

Trace length over eight fixed probes (mean tokens between the reasoning tags):

The SFT row is not a typo. At 8B, supervised fine-tuning on a mixture dominated by another language's reasoning tokens made the model emit an empty reasoning block on 64.5% of generations — the empty <think></think> pair is a near-deterministic two-token continuation and became a low-loss attractor. DPO restored the block; RLVR restored the language.

Benchmark results

HKCanto-Eval (Cheng et al., 2025), reasoning mode on:

Read this table carefully. SFT cost 20.52 points. DPO recovered 10.19 and RLVR a further 3.62, leaving the model 6.71 points below the merged checkpoint it started from. The 30B-A3B closed the same gap to 1.20 points; this model did not. What it gained instead is Cantonese reasoning, which this benchmark does not measure.

The paper's stated reason is that teaching a model to reason in a new language needs Cantonese reasoning traces present during continual pre-training, not only in post-training — and the 8B had the least headroom to absorb that gap.

If you can run the 30B-A3B, run [that one](https://huggingface.co/hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking) instead. This model is released for deployments where 8B dense is the constraint, and because the contrast between the two sizes is one of the paper's findings.

Artefacts released with this work

Every post-training stage on this card is reproducible from public components. The CPT corpus itself is not released, but its two largest public constituents are.

The three-way parallel structure of the RL corpus is what makes the cross-language pass-rate gap reported in the paper measurable: for the translated environments, prompt language is the only variable.

Other models in this release

All four checkpoints are in the CantoneseLLM v2.0 collection.

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95)
print(tokenizer.decode(out[0][len(inputs.input_ids[0]):], skip_special_tokens=True))

vLLM

The stock command works — no special flags are required:

vllm serve hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking

Add --tensor-parallel-size N to shard across N GPUs, and --reasoning-parser qwen3 if you want the reasoning block returned separately as reasoning_content rather than inline in content (parser names vary by vLLM version).

OpenAI-compatible API

The served endpoint speaks the OpenAI protocol, so the official client works unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"

resp = client.chat.completions.create(
    model="hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking",
    messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
    ],
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048,
)

msg = resp.choices[0].message
print(getattr(msg, "reasoning_content", None) or "")  # populated with --reasoning-parser
print(msg.content)

Or with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking",
    "messages": [
      {"role": "system", "content": "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"},
      {"role": "user", "content": "點解香港嘅雨季集中喺五月到九月？"}
    ],
    "temperature": 0.6, "top_p": 0.95, "max_tokens": 2048
  }'

Sampling: temperature 0.6, top-p 0.95 — the settings used for every evaluation reported above and in the paper. Avoid greedy decoding.

System prompt: the Cantonese system prompt above was used throughout training and evaluation. Behaviour with other system prompts, or with none, is not characterised.

Thinking and non-thinking. Unlike the 30B-A3B, 25% of this model's SFT data carried no chain-of-thought, deliberately, so that it retains the ability to answer without an explicit reasoning block. It is the closer of the two to stock Qwen3 hybrid behaviour — but given the empty-block history above, verify the mode switch on your own prompts rather than assuming it.

Training pipeline

Released weights are the final step of stage 2, not a best-validation checkpoint.

Risks & Limitations

• —Does not fully recover pre-SFT benchmark performance — 6.71 points below the merged checkpoint. See the table above. The 30B-A3B closed this gap; this model did not.
• —Short reasoning traces. 232 tokens on average, at a CoT-to-answer ratio of 1.09. No long original Cantonese reasoning traces existed at the scale SFT needed; human-written or human-verified traces amounted to 131 rows. Most Cantonese CoT in training was machine-translated from English or Simplified Chinese.
• —History of empty reasoning blocks. The SFT checkpoint emitted an empty block on 64.5% of generations. DPO and RLVR repaired this, but if you fine-tune further from here, watch for the attractor returning.
• —No Hong Kong grounding in the reasoning data. The SFT mixture contained no content grounded in Hong Kong entities or current events — that knowledge comes from CPT only.
• —Benchmark scores are underestimates of knowledge. DPO's pair-selection weighted longer responses with no reward for instruction compliance, so the model sometimes ignores "answer with the letter only". Multiple-choice parsers that read a bare leading option letter will under-score it. Use a marker-anchored extractor.
• —One run per stage. Severe compute constraints meant no hyperparameter sweep in any post-training stage.
• —Preference data were model-judged, not human-annotated, and are bounded by the judge's own Cantonese ability.
• —Long context is inherited and untested. Training sequence lengths were 8,192 (SFT/DPO) and 16,384 (RLVR stage 2). Behaviour beyond that is whatever the Qwen3 base provides.
• —Standard LLM caveats apply: it will hallucinate, and it has not been safety- tuned beyond what the chat vector carried over.

Citation

@misc{cantonesellm_v2,
      title={CantoneseLLM v2: Reasoning in a Low-Resource Language},
      author={Tsz Chung Cheng and Chung Shing Cheng and Chaak Ming Lau and Cheuk Hei Chong},
      year={2026},
      eprint={2609.06970},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.06970},
}

Acknowledgements

Continual pre-training (CPT) was carried out on Cloud TPUs (Tensor Processing Units) from Google's TPU Research Cloud with MaxText. Post-training was carried out on computer resources offered under the category of General Projects by Research Institute for Information Technology, Kyushu University. Usage fee and cost of data-curation costs with proprietary APIs were covered by Votee AI

The RLVR stage builds on NVIDIA's NeMo-RL and NeMo-Gym, and on the Nemotron post-training and RL datasets.