AxionLab-official/MiniBot-0.9M-Base

🧠 MiniBot-0.9M-Base

Ultra-lightweight GPT-2 style language model (~985K parameters) specialized in Portuguese conversational text.

   

📌 Overview

MiniBot-0.9M-Base is a tiny decoder-only Transformer (~985K parameters) based on the GPT-2 architecture, designed for efficient text generation in Portuguese.

This is a base (pretrained) model — trained purely for next-token prediction, with no instruction tuning or alignment of any kind. It serves as the foundation for fine-tuned variants such as MiniBot-0.9M-Instruct.

🎯 Key Characteristics

🏗️ Architecture

MiniBot-0.9M follows a scaled-down GPT-2 design:

• —Token embeddings + positional embeddings
• —Multi-head self-attention
• —Feed-forward (MLP) layers
• —Autoregressive decoding

Despite its small size, it preserves the core inductive biases of GPT-2, making it well-suited for experimentation and educational purposes.

📚 Training Dataset

The model was trained on a Portuguese conversational dataset focused on language pattern learning.

Training notes:

• —Pure next-token prediction objective
• —No instruction tuning (no SFT, no RLHF, no alignment)
• —Lightweight training pipeline
• —Optimized for small-scale experimentation

💡 Capabilities

✅ Strengths

• —Portuguese text generation
• —Basic dialogue structure
• —Simple prompt continuation
• —Linguistic pattern learning

❌ Limitations

• —Very limited reasoning ability
• —Loses context in long conversations
• —Inconsistent outputs
• —Prone to repetition or incoherence

⚠️ This model behaves as a statistical language generator, not a reasoning system.

🚀 Getting Started

Installation

pip install transformers torch

Usage with Hugging Face Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "AxionLab-official/MiniBot-0.9M-Base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "User: Me explique o que é gravidade\nBot:"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.8,
    top_p=0.95,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚙️ Recommended Settings

💡 Base models generally benefit from higher temperature values compared to instruct variants, since there is no fine-tuning to constrain the output distribution.

🧪 Intended Use Cases

⚠️ Disclaimer

• —Extremely small model (~985K parameters)
• —Limited world knowledge and weak generalization
• —No safety or alignment measures
• —Not suitable for production use

🔮 Future Work

• —[x] 🎯 Instruction-tuned version → `MiniBot-0.9M-Instruct`
• —[ ] 📚 Larger and more diverse dataset
• —[ ] 🔤 Tokenizer improvements
• —[ ] 📈 Scaling to 1M–10M parameters
• —[ ] 🧠 Experimental reasoning fine-tuning

📜 License

Distributed under the MIT License. See `LICENSE` for more details.

👤 Author

Developed by [AxionLab](https://huggingface.co/AxionLab-official) 🔬

<div align="center"> _{MiniBot-0.9M-Base · AxionLab · MIT License} </div>