Jaume-inLab/vocabulary_sliced_CA-ES-EN-qwen3-14B

Qwen3-14B Vocabulary Sliced (CA-ES-EN)

This model is a vocabulary-sliced version of the original Qwen/Qwen3-14B. The vocabulary has been pruned to retain only tokens relevant to Catalan, Spanish, and English, as well as essential special tokens and base bytes.

Motivation

The primary goal of this pruning is to reduce the memory footprint (VRAM) during training and inference, specifically for tasks like Direct Preference Optimization (DPO) with long sequence lengths.

By reducing the vocabulary size from 152,064 to ~134,943 tokens, the size of the embed_tokens and lm_head matrices is significantly reduced. This provides:

• —Theoretical VRAM Savings: ~1.06 GB (assuming 16-bit precision for embeddings/lm_head).
• —Real VRAM Savings: ~1.90 GB during forward passes with long sequences (e.g., 8192 tokens), due to the reduced size of the logits tensor before the softmax operation.

Compatibility

This pruned model is 100% compatible with LoRA adapters trained on the original Qwen/Qwen3-14B model, provided the adapters only target the Attention and MLP layers (e.g., q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).

Because the adapter weights do not interact with the embed_tokens or lm_head matrices, they can be attached to this pruned base model without any shape mismatches or degradation in output quality.

Pruning Methodology

The vocabulary was pruned using the following criteria:

1. 1.Dataset Tokens: All tokens present in a specific corpus of Catalan, Spanish, and English text were kept.
2. 2.Special Tokens: All special tokens (IDs >= 151643) were kept.
3. 3.Base Bytes: All base byte tokens (IDs 0-255) were kept.
4. 4.Allowed Characters: Any token consisting entirely of a predefined set of allowed characters (ASCII + specific Catalan/Spanish characters like áéíóúüñÁÉÍÓÚÜÑ¿¡àèòïçÀÈÒÏÇ·€) was kept.
5. 5.BPE Merge Tree Repair: Parent tokens were iteratively added back to the vocabulary to ensure the BPE merge rules remained valid and did not cause fragmentation.

Usage

You can load this model just like the original Qwen3-14B model:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Jaume-inLab/vocabulary_sliced_CA-ES-EN-qwen3-14B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")