Modelpublic

Dummy9898/bear-240m-pretrain

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes36downloads

Model Card

🐻 Mesosfer Bear AI - PRETRAIN

Mesosfer Bear AI (241.8M) is a high-efficiency autoregressive decoder-only language model built on a Llama-style architecture. This repository contains the official model weights and runtime engine for PRETRAIN ().

🌟 Model Architecture Highlights

—Parameters: 241,828,864 (241.8M)
—Layers / Depth: 16 transformer blocks
—Hidden Dimension (`d_model`): 1024
—FFN Hidden Dimension: 2816 (SwiGLU activation)
—Attention Heads: 16 Query heads / 4 KV heads (Grouped Query Attention 4:1)
—Context Length: 4096 tokens
—Positional Encoding: Rotary Position Embeddings (RoPE, $\theta=10000$)
—Tokenizer: 60,000 vocabulary based on Kimi-K3 BPE with native XTML markup (<|open|>...<|close|>) and Rust tiktoken acceleration.
—Training Step: Step 12,000 (Loss: 1.8174)

🚀 Quickstart: Running Inference

You can run text generation and chat streaming immediately with zero external frameworks:

1. Installation

bash

git clone https://huggingface.co/{REPO_ID}
cd {REPO_NAME}
pip install torch tiktoken

2. Standalone Inference Script

bash

python inference.py --prompt "Jelaskan konsep machine learning secara singkat:"

3. Interactive Streaming Chat CLI

bash

python cli.py --temperature 0.7 --top-p 0.9

4. Python API Usage

python

from engine.transformer import BearTransformer, BearConfig
from engine.tokenizer import BearTokenizer
import torch

# Load Tokenizer & Model
tokenizer = BearTokenizer.load("bear_tokenizer.json")
config = BearConfig.from_dict(torch.load("config.json"))
model = BearTransformer(config)

checkpoint = torch.load("bear_model.pt", map_location="cuda" if torch.cuda.is_available() else "cpu")
model.load_state_dict(checkpoint["model_state"] if "model_state" in checkpoint else checkpoint)
model.eval()

# Chat format
conversation = [
    {"role": "system", "content": "Anda adalah asisten AI Bear yang cerdas dan ramah."},
    {"role": "user", "content": "Halo! Siapa kamu?"}
]
prompt = tokenizer.apply_chat_template(conversation, thinking=True)
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

output_ids = model.generate(input_ids, max_new_tokens=256, temperature=0.7, top_p=0.9)
response = tokenizer.decode(output_ids[0].tolist())
print(response)

📜 License

Distributed under the Apache-2.0 License. Developed by Mesosfer Team.