AxionLab-official/MiniBot-0.9M-Instruct

🧠 MiniBot-0.9M-Instruct

Instruction-tuned GPT-2 style language model (~900K parameters) optimized for Portuguese conversational tasks.

   

📌 Overview

MiniBot-0.9M-Instruct is the instruction-tuned version of MiniBot-0.9M-Base, designed to follow prompts more accurately, respond to user inputs, and generate more coherent conversational outputs in Portuguese.

Built on a GPT-2 architecture (~0.9M parameters), this model was fine-tuned on conversational and instruction-style data to improve usability in real-world interactions.

🎯 Key Characteristics

🧠 What Changed from Base?

Instruction tuning introduced significant behavioral improvements with no architectural changes:

💡 The model is now significantly more usable in chat scenarios.

🏗️ Architecture

The core architecture remains identical to the base model:

• —Decoder-only Transformer (GPT-2 style)
• —Token embeddings + positional embeddings
• —Self-attention + MLP blocks
• —Autoregressive generation

No structural changes were made — only behavioral improvement through fine-tuning.

📚 Fine-Tuning Dataset

The model was fine-tuned on a Portuguese instruction-style conversational dataset composed of:

• —💬 Questions and answers
• —📋 Simple instructions
• —🤖 Assistant-style chat
• —🎭 Basic roleplay
• —🗣️ Natural conversations

Expected format:

User: Me explique o que é gravidade
Bot: A gravidade é a força que atrai objetos com massa...

Training strategy:

• —Supervised Fine-Tuning (SFT)
• —Pattern learning for instruction-following
• —No RLHF or preference optimization

💡 Capabilities

✅ Strengths

• —Following simple instructions
• —Answering basic questions
• —Conversing more naturally
• —Higher coherence in short responses
• —More consistent dialogue structure

❌ Limitations

• —Reasoning is still limited
• —May generate incorrect facts
• —Does not retain long context
• —Sensitive to poorly structured prompts

⚠️ Even with instruction tuning, this remains an extremely small model. Adjust expectations accordingly.

🚀 Getting Started

Installation

pip install transformers torch

Usage with Hugging Face Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "AxionLab-official/MiniBot-0.9M-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "User: Me diga uma curiosidade sobre o espaço\nBot:"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚙️ Recommended Settings

💡 Instruct models tend to perform better at lower temperatures. Try values around 0.65 for more accurate and focused responses.

🧪 Intended Use Cases

⚠️ Disclaimer

• —Extremely small model (~900K parameters)
• —No robust alignment (no RLHF)
• —May generate incorrect or nonsensical responses
• —Not suitable for critical production environments

🔮 Future Work

• —[ ] 🧠 Reasoning-tuned version (MiniBot-Reason)
• —[ ] 📈 Scaling to 1M–10M parameters
• —[ ] 📚 Larger and more diverse dataset
• —[ ] 🤖 Improved response alignment
• —[ ] 🧩 Tool-use experiments

📜 License

Distributed under the MIT License. See `LICENSE` for more details.

👤 Author

Developed by [AxionLab](https://huggingface.co/AxionLab-official) 🔬

<div align="center"> _{MiniBot-0.9M-Instruct · AxionLab · MIT License} </div>