AxionLab-official/MiniBot-0.9M-Base

1---2license: mit3language:4- pt5pipeline_tag: text-generation6tags:7- base8- pretrain9- pretrained10- nano11- mini12- chatbot13library_name: transformers14---15 16# 🧠 MiniBot-0.9M-Base17 18> **Ultra-lightweight GPT-2 style language model (~985K parameters) specialized in Portuguese conversational text.**19 20[![Model](https://img.shields.io/badge/🤗%20Hugging%20Face-MiniBot--0.9M--Base-yellow)](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Base)21[![License](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)22[![Language](https://img.shields.io/badge/Language-Portuguese-blue)](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Base)23[![Parameters](https://img.shields.io/badge/Parameters-~985K-orange)](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Base)24 25---26 27## 📌 Overview28 29**MiniBot-0.9M-Base** is a tiny decoder-only Transformer (~985K parameters) based on the GPT-2 architecture, designed for efficient text generation in **Portuguese**.30 31This is a **base (pretrained) model** — trained purely for next-token prediction, with no instruction tuning or alignment of any kind. It serves as the foundation for fine-tuned variants such as [MiniBot-0.9M-Instruct](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Instruct).32 33---34 35## 🎯 Key Characteristics36 37| Attribute | Detail |38|---|---|39| 🇧🇷 **Language** | Portuguese (primary) |40| 🧠 **Architecture** | GPT-2 style (Transformer decoder-only) |41| 🔤 **Embeddings** | GPT-2 compatible |42| 📉 **Parameters** | ~985K (985,728) |43| ⚙️ **Objective** | Causal Language Modeling (next-token prediction) |44| 🚫 **Alignment** | None (base model) |45 46---47 48## 🏗️ Architecture49 50MiniBot-0.9M follows a scaled-down GPT-2 design:51 52- Token embeddings + positional embeddings53- Multi-head self-attention54- Feed-forward (MLP) layers55- Autoregressive decoding56 57Despite its small size, it preserves the core inductive biases of GPT-2, making it well-suited for experimentation and educational purposes.58 59---60 61## 📚 Training Dataset62 63The model was trained on a Portuguese conversational dataset focused on language pattern learning.64 65**Training notes:**66- Pure next-token prediction objective67- No instruction tuning (no SFT, no RLHF, no alignment)68- Lightweight training pipeline69- Optimized for small-scale experimentation70 71---72 73## 💡 Capabilities74 75### ✅ Strengths76 77- Portuguese text generation78- Basic dialogue structure79- Simple prompt continuation80- Linguistic pattern learning81 82### ❌ Limitations83 84- Very limited reasoning ability85- Loses context in long conversations86- Inconsistent outputs87- Prone to repetition or incoherence88 89> ⚠️ This model behaves as a statistical language generator, not a reasoning system.90 91---92 93## 🚀 Getting Started94 95### Installation96 97```bash98pip install transformers torch99```100 101### Usage with Hugging Face Transformers102 103```python104from transformers import AutoTokenizer, AutoModelForCausalLM105 106model_name = "AxionLab-official/MiniBot-0.9M-Base"107tokenizer = AutoTokenizer.from_pretrained(model_name)108model = AutoModelForCausalLM.from_pretrained(model_name)109 110prompt = "User: Me explique o que é gravidade\nBot:"111inputs = tokenizer(prompt, return_tensors="pt")112 113outputs = model.generate(114    **inputs,115    max_new_tokens=50,116    temperature=0.8,117    top_p=0.95,118    do_sample=True,119)120 121print(tokenizer.decode(outputs[0], skip_special_tokens=True))122```123 124### ⚙️ Recommended Settings125 126| Parameter | Recommended Value | Description |127|---|---|---|128| `temperature` | `0.7 – 1.0` | Controls randomness |129| `top_p` | `0.9 – 0.95` | Nucleus sampling |130| `do_sample` | `True` | Enable sampling |131| `max_new_tokens` | `30 – 80` | Response length |132 133> 💡 Base models generally benefit from higher temperature values compared to instruct variants, since there is no fine-tuning to constrain the output distribution.134 135---136 137## 🧪 Intended Use Cases138 139| Use Case | Suitability |140|---|---|141| 🧠 Fine-tuning (chat, instruction, roleplay) | ✅ Ideal |142| 🎮 Prompt playground & experimentation | ✅ Ideal |143| 🔬 Research on tiny LLMs | ✅ Ideal |144| 📉 Benchmarking small architectures | ✅ Ideal |145| ⚡ Local / CPU-only applications | ✅ Ideal |146| 🏭 Critical production environments | ❌ Not recommended |147 148---149 150## ⚠️ Disclaimer151 152- Extremely small model (~985K parameters)153- Limited world knowledge and weak generalization154- No safety or alignment measures155- **Not suitable for production use**156 157---158 159## 🔮 Future Work160 161- [x] 🎯 Instruction-tuned version → [`MiniBot-0.9M-Instruct`](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Instruct)162- [ ] 📚 Larger and more diverse dataset163- [ ] 🔤 Tokenizer improvements164- [ ] 📈 Scaling to 1M–10M parameters165- [ ] 🧠 Experimental reasoning fine-tuning166 167---168 169## 📜 License170 171Distributed under the **MIT License**. See [`LICENSE`](LICENSE) for more details.172 173---174 175## 👤 Author176 177Developed by **[AxionLab](https://huggingface.co/AxionLab-official)** 🔬178 179---180 181<div align="center">182  <sub>MiniBot-0.9M-Base · AxionLab · MIT License</sub>183</div>