nassimjp/MiniCPM5-2B-SFT-Pashto

🧬 MiniCPM5-2B-Pashto

Pashto + Urdu Tokenizer Surgery Model

📋 Model Overview

This is a tokenizer-modified version of openbmb/MiniCPM5-2B with 46 Pashto and Urdu-specific characters added as single tokens to the vocabulary.

🔍 Why This Model Exists

MiniCPM5-2B's tokenizer does not natively support many Pashto and Urdu characters as single tokens. This causes:

• —❌ Characters being split into multiple tokens
• —❌ Inefficient encoding
• —❌ Poor representation for Pashto/Urdu text

This model fixes that by surgically adding 46 missing atoms as independent tokens.

📊 Complete Audit & Modification Report

✅ Existing Single-Token Atoms (20)

⚠️ Split Atoms — FIXED via Surgery (46)

🔬 Embedding Forensics

Model Configuration

{
    "architectures": ["LlamaForCausalLM"],
    "vocab_size": 130606,
    "hidden_size": 2048,
    "num_hidden_layers": 42,
    "max_position_embeddings": 131072,
    "tie_word_embeddings": False,
    "initializer_range": 0.02,
}

Input Embedding Statistics (New Atoms)

Most Similar Pair (New Atoms)

✅ No exact duplicate embeddings detected

🚀 Usage

Installation

pip install transformers torch accelerate

Basic Loading

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model and tokenizer
model_id = "nassimjp/MiniCPM5-2B-Pashto"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Tokenization Check

# Test Pashto characters
pashto_text = "سلام ښه راغلئ"

# Tokenize
tokens = tokenizer.encode(pashto_text, add_special_tokens=False)
print(f"Token IDs: {tokens}")

# Decode back
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")

# Check individual atoms
for char in "ښڅځڼږډټړېۍګ":
    ids = tokenizer.encode(char, add_special_tokens=False)
    print(f"'{char}' → {ids}")

Text Generation

# Simple generation
prompt = "ښه راغلئ"
inputs = tokenizer.encode(prompt, return_tensors="pt")

# Generate
with torch.no_grad():
    outputs = model.generate(
        inputs,
        max_new_tokens=50,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚙️ Technical Details

Surgery Methodology

1. 1.Audit Phase
2. 2.Iterated through 66 Pashto/Urdu characters
3. 3.Detected which characters split into multiple tokens
4. 4.Identified 46 characters needing surgery

1. 1.Token Addition
2. 2.Used tokenizer.add_tokens() to add 46 missing atoms
3. 3.Resized model embeddings with model.resize_token_embeddings()
4. 4.New tokens assigned IDs: 130560 → 130605

1. 1.Independent Initialization
2. 2.Input embeddings initialized with normal distribution
3. 3.Output embeddings initialized independently
4. 4.Both use initializer_range = 0.02
5. 5.Not tied — maintains separate input and output spaces

1. 1.Verification
2. 2.Tokenization gate confirmed all 66 atoms as single tokens
3. 3.No exact duplicate embeddings detected
4. 4.Embedding norms and covariances verified

⚠️ Important Notes

📦 Model Files

MiniCPM5-2B-Pashto/
├── chat_template.jinja             0.01 MB
├── config.json                     0.00 MB
├── generation_config.json          0.00 MB
├── ipashto_surgery_info.txt        0.00 MB
├── model-00001-of-00003.safetensors  1,903.45 MB
├── model-00002-of-00003.safetensors  1,907.19 MB
├── model-00003-of-00003.safetensors    990.10 MB
├── model.safetensors.index.json    0.03 MB
├── tokenizer.json                  9.44 MB
└── tokenizer_config.json           0.00 MB

Total Size: 4.70 GB

🔗 Related Resources

• —Base Model: openbmb/MiniCPM5-2B
• —Original Paper: MiniCPM: Unveiling the Potential of Small Language Models
• —GitHub: OpenBMB/MiniCPM

📝 Citation

If you use this model, please cite the original MiniCPM paper:

@article{minicpm2024,
  title={MiniCPM: Unveiling the Potential of Small Language Models},
  author={Hu, Shengding and Ding, Ning and others},
  journal={arXiv preprint arXiv:2404.06395},
  year={2024}
}

📄 License

This model is released under the Apache License 2.0.

🤝 Acknowledgements

• —OpenBMB for developing and releasing MiniCPM5-2B
• —Hugging Face for the transformers library and model hosting
• —Kaggle for providing the compute environment

🧪 Testing Results

All 66 Pashto/Urdu atoms successfully tokenize as single tokens:

Created with ❤️ for the Pashto and Urdu communities

---