PredictiveManish/Trimurti-LM

1---2---3license: apache-2.04tags:5- multilingual6- text-generation7- indic-languages8- hindi9- punjabi10- small-model11pipeline_tag: text-generation12widget:13- text: "[EN] The weather today is"14  example_title: "English Generation"15- text: "[HI] आज का मौसम"16  example_title: "Hindi Generation"17- text: "[PA] ਅੱਜ ਦਾ ਮੌਸਮ"18  example_title: "Punjabi Generation"19language:20- en21- hi22- pa23datasets:24- ai4bharat/samanantar25- PredictiveManish/multilingual-corpus26library_name: transformers27---28 29# Trimurti-LM: A 4.2M Parameter Multilingual Language Model30 31## Model Description32 33**Trimurti-LM** is a small, efficient multilingual language model trained from scratch on English, Hindi, and Punjabi text. Named after the Hindu trinity (Brahma-Vishnu-Shiva), it represents the three-fold capability of creating text, preserving meaning, and transforming across scripts.34 35**Key Features:**36- 🏗️ **Built from scratch** - No pre-trained weights used37- 🌐 **Multilingual** - Handles 3 languages with 3 different scripts38- 💾 **Tiny footprint** - Only 4.2 million parameters39- ⚡ **Fast training** - 2.38 hours on consumer GPU (GTX 1650 4GB)40- 🔤 **Smart tokenization** - Custom SentencePiece with byte fallback for Indic scripts41 42## Model Specifications43 44| Aspect | Details |45|--------|---------|46| **Architecture** | GPT-2 style decoder-only Transformer |47| **Parameters** | 4,672,000 (4.2M) |48| **Hidden Size** | 256 |49| **Layers** | 4 |50| **Attention Heads** | 8 |51| **Context Length** | 128 tokens |52| **Vocabulary** | 8000 tokens (SentencePiece) |53| **Training Steps** | 5000 |54| **Training Time** | 2.38 hours |55| **Hardware** | NVIDIA GTX 1650 (4GB VRAM) |56 57## Training Data58 59The model was trained on a balanced multilingual corpus:60- **English**: 150,000 sentences61- **Hindi**: 150,000 sentences  62- **Punjabi**: 150,000 sentences63 64**Sources:**65- Primary: AI4Bharat Samanantar dataset (filtered and processed)66- Secondary: Custom curated multilingual corpus67 68**Data Processing:**69- Language tagging: `[EN]`, `[HI]`, `[PA]` prefixes70- Length filtering: 5-50 words per sentence71- Script validation for each language72- Deduplication and cleaning73 74## Performance75 76| Metric | Value | Notes |77|--------|-------|-------|78| **Final Loss** | 1.206 | Cross-entropy loss |79| **Perplexity** | 3.32 | e^1.206 = 3.32 |80| **Top-1 Accuracy** | ~25% | Next token prediction |81| **Top-5 Accuracy** | ~60% | Next token prediction |82| **Language ID Accuracy** | 95% | With explicit tags |83 84## Usage85 86### Quick Start87 88```python89from transformers import GPT2LMHeadModel90import sentencepiece as spm91import torch92 93# Load model and tokenizer94tokenizer = spm.SentencePieceProcessor()95tokenizer.load("multilingual_spm.model")96model = GPT2LMHeadModel.from_pretrained("PredictiveManish/Trimurti-LM")97 98# Generate text99prompt = "[EN] The weather is"100input_ids = tokenizer.encode(prompt)101input_tensor = torch.tensor([input_ids])102 103with torch.no_grad():104    output = model.generate(105        input_ids=input_tensor,106        max_length=50,107        temperature=0.7,108        do_sample=True,109        pad_token_id=0110    )111 112generated = tokenizer.decode(output[0].tolist())113print(generated)114 115 116```117 118## citations(surely you're not going to use this but still, if in search of worst models): 119If you use Trimurti-LM in your work, please cite:120 121```bibtex122@software{trimurti_lm_2026,123  title = {Trimurti-LM: A 4.2M Parameter Multilingual Language Model},124  author = {Manish Tiwari},125  year = {2026},126  url = {https://huggingface.co/PredictiveManish/Trimurti-LM},127  note = {Trained from scratch on English, Hindi, and Punjabi with consumer hardware}128}129 130 131```132 133 134### Primary Dataset135 136```bibtex137@inproceedings{samanantar_2021,138  title = {Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages},139  author = {Gowtham Ramesh and Sumanth Doddapaneni and Aravinth Bheemaraj and Mayank Jobanputra and Raghavan AK and Ajitesh Sharma and Sujit Sahoo and Harshita Diddee and Mahalakshmi J and Divyanshu Kakwani and Navneet Kumar and Aswin Pradeep and Srihari Nagaraj and Kumar Deepak and Vivek Raghavan and Anoop Kunchukuttan and Pratyush Kumar and Mitesh Shantadevi Khapra},140  booktitle = {Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks},141  year = {2021},142  url = {https://arxiv.org/abs/2104.05596}143}144```145---