Se00n00/TinyLM2-50M-IFT
TinyLM2-50M-Instruct
TinyLM2-50M-Instruct is a compact decoder-only Transformer language model designed for efficient instruction following and conversational AI. The model has approximately 50M parameters and has been fine-tuned using Supervised Fine-Tuning (SFT) on the UltraChat 200k instruction-response dataset to improve chat capabilities while maintaining a lightweight footprint suitable for local inference and resource-constrained environments.
Evaluation
All evaluations are zero-shot unless stated otherwise, and i used lmeval to run them <img src="losschart.png">
Model Architecture & Hyperparameters
TinyLM2-50M-Instruct is built on a custom ALiBi Decoder-Only Transformer architecture with pre-normalization and gated feedforward networks:
## Tokenizer & Chat Template
The model uses a custom Byte-Level BPE Tokenizer equipped with special tokens and a pre-configured Jinja2 chat_template for multi-turn conversations.
## Training Configuration
## Inference
# pip install torch transformers
import torch
from transformers import pipeline
pipe = pipeline(
"text-generation",
model="Se00n00/TinyLM2-50M-Instruct",
trust_remote_code = True
)
messages = [
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "Explain artificial intelligence in simple terms."}
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
result = pipe(
prompt,
max_new_tokens=120,
do_sample=True,
temperature=0.5,
top_k=40,
top_p=0.9
)
print(result[0]['generated_text']) ──────
Sample Outputs
Chat-templated generation (temperature 0.8, top-p 0.9):
System: You are a helpful AI assistant
User: Write a short poem about the ocean.
The ocean is alive with life, As it floats in the waves, A vast expanse of vastness, A sight that cannot be matched, A wonderland that awakens, A place where the sun shines, The sea, it sets. The ocean is a reminder of life, The endless ocean, a place of peace, A place where the sun shines, The ocean is a beacon of hope, A place where the ocean is alive …
System: You are a helpful AI assistant
User: Explain the concept of gravity to a child.
The concept of gravity to a child is an important aspect of the child's development. It is the foundation of their physical and mental health. When a child is in a state of gravity, they are not able to move or move objects, which can lead to mental health problems such as anxiety, depression, and depression. The concept of gravity to a child is a crucial aspect of their development. It helps them to develop their mental health and develop their ability to control their own gravity. One of the main benefits of gravity is that …