Modelpublic

chaitanyaapex98/nanogpt-fineweb-edu

sourceHugging Facemitupdated 5mo agoView on Hugging Face

2likes

Model Card

nanogpt-sft-finance

A GPT-style language model trained from scratch with finance specialisation. Built using the nanogpt pipeline.

Model Details

Property	Value
Architecture	GPT (RoPE, GQA, RMSNorm, Flash Attention 3, Muon optimizer)
Checkpoint step	046831
Checkpoint file	`model_046831.pt`
Tokenizer	Custom BPE (`tokenizer.pkl` + `token_bytes.pt`)

Training Data

Pretraining: FineWeb-Edu + SEC 10-K filings + Python code (CodeParrot)

SFT fine-tuning:

—SmolTalk (~267K general conversations)
—Finance-Alpaca (~68K finance Q&A)
—GPT-4o mini CoT traces on ConvFinQA problems
—GPT-4o mini finance Python code examples

How to Use

This model uses a custom tokenizer and architecture — it does not work with AutoModel.from_pretrained(). Clone the repository and use chat_cli.py:

bash

git clone https://github.com/ChaitanyaPandit1998/nanogpt
cd nanogpt
git checkout feature/nanogpt-2.0
pip install -r requirements.txt

# Download model files from this HuggingFace repo
hf download chaitanyaapex98/nanogpt-sft-finance --local-dir ./hf_model/

# Run interactive chat
python chat_cli.py \
  --model-dir     ./hf_model/ \
  --tokenizer-dir ./hf_model/tokenizer/

Limitations

—Finance formulas (Sharpe ratio, EV/EBITDA) are partially learned but not always precise
—Best results on questions grounded in provided context (SEC filing text)
—Run 2 with shuffled training data is in progress — expect improved formula recall

Repository

ChaitanyaPandit1998/nanogpt — feature/nanogpt-2.0 branch