wejyy888/Qwen3.5-0.8B-PureText-ZH-EN

Qwen3.5-0.8B Pure Text ZH-EN

A text-only derivative of Qwen/Qwen3.5-0.8B optimized for Chinese and English inference.

Changes

• —Converted Qwen3_5ForConditionalGeneration to Qwen3_5ForCausalLM.
• —Removed the vision tower and MTP weights.
• —Removed visual, grounding, audio, and TTS special tokens.
• —Pruned the vocabulary from 248,320 rows to 182,684 real tokens, padded to 182,784 rows for alignment.
• —Remapped tokenizer IDs and retained old2new.json for converting existing tokenized datasets.
• —Preserved the original weights exactly for every retained vocabulary item; no additional training was performed.

The resulting checkpoint has 685,284,160 parameters and occupies approximately 1.3 GiB in BF16 safetensors format.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "wejyy888/Qwen3.5-0.8B-PureText-ZH-EN"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")

messages = [{"role": "user", "content": "介绍一下杭州"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use a Transformers version that includes Qwen3_5ForCausalLM support.

Compatibility notes

• —Previously tokenized datasets must be remapped with old2new.json.
• —Chinese and English tokenization is preserved exactly after ID remapping.
• —Other languages still decode correctly through byte fallback, but generally use more tokens.
• —This checkpoint accepts text only; image, video, audio, and TTS inputs are unsupported.
• —A short continued-pretraining or SFT stage is recommended before production use to adapt to the reduced vocabulary.

Reproduction

prune_qwen35_zh_en.py contains the conversion procedure. The padded vocabulary rows are initialized to zero so they cannot produce invalid high logits.

Validation

• —Loaded successfully through AutoModelForCausalLM as Qwen3_5ForCausalLM.
• —No visual or MTP parameters remain.
• —Retained-token logits match the source model exactly in the comparison test.
• —Chinese/English tokenizer equivalence, text chat-template encoding, forward pass, and generation smoke tests pass.