Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens

Bharat MiniGPT 350M (3.5B tokens Experiment)

Bharat MiniGPT 350M is a custom GPT-style causal language model trained from scratch by Harshvardhan Mishra using modern LLM architecture components such as RoPE, RMSNorm, SwiGLU, and SDPA Attention.

This is not a fine-tuned GPT-2 or LLaMA variant. The architecture and training pipeline were implemented manually in PyTorch and later integrated into the HuggingFace ecosystem.

A better version with more tokens and fine tune version available SOON.

Best suited for:

Knowledge completion Educational prompts Article continuation English prose generation

This is a pretrained foundation model and is not instruction tuned yet.

Explore More:https://iotbyhvm.ooo/bharat-minigpt-350m-a-custom-gpt-style-llm-built-from-scratch-in-india/

Model Details

• —Model Name: Bharat MiniGPT 350M
• —Parameters: ~350 Million
• —Architecture: Decoder-only Transformer
• —Training Tokens: 3.5 Billion
• —Framework: PyTorch + Custom Hugging Face Transformers integration
• —Developer: Harshvardhan Mishra
• —Organization: HVM Smart Solutions

Architecture

Training Data

The model was trained using a weighted mixture of:

Training Setup

Features

• —Custom GPT architecture
• —RoPE positional embeddings
• —RMSNorm normalization
• —SwiGLU feed-forward layers
• —Flash Attention compatible SDPA
• —HuggingFace generate() support
• —KV-cache compatible
• —Weight tying support
• —Gradient checkpointing during training

Benchmark Results

Evaluated using: EleutherAI LM Evaluation Harness

Notes:

• —Results are from the current 3B tokens pretrained base checkpoint.
• —This model is not instruction-tuned yet.
• —Further tokenizer and training improvements are planned.

Installation

pip install transformers torch

Usage

Load Model from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.frompretrained( "Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens", trustremote_code=True )

tokenizer = AutoTokenizer.frompretrained( "Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens", trustremote_code=True )

Generate Text

import torch

prompt = "India is a land of"

inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad(): outputs = model.generate( **inputs,

maxnewtokens=80,

temperature=0.45,

top_p=0.82,

top_k=40,

repetition_penalty=1.35,

norepeatngram_size=4,

do_sample=True,

use_cache=True,

eostokenid=tokenizer.eostokenid, padtokenid=tokenizer.eostokenid, )

print(tokenizer.decode(output[0], skipspecialtokens=True))