Modelpublic

ibnsina-llm/ibnsina-3b-checkpoints

sourceHugging Faceapache-2.0updated 16d agoView on Hugging Face

0likes158downloads

Model Card

ibnsina-3b-checkpoints — raw training checkpoints of IbnSina-3B

These are not the files you want for running the model. For that, use ibnsina-llm/ibnsina-3b (GGUF for llama.cpp / ollama / LM Studio: ollama run ibnsina/ibnsina-3b).

This repository holds the raw [nanochat](https://github.com/karpathy/nanochat)-format checkpoints (.pt, PyTorch pickles of tensors) of the IbnSina-3B pretraining and fine-tuning runs, published for reproducibility, research and fine-tuning from the base. Architecture: Qwen3-style (40 layers, d=2560, 20 heads / 4 KV heads, QK-norm), 3.0 B parameters, 2048 context, tokenizer v2_32k_llama (32,768 BPE; shared with IbnSina-1.5B). Loading requires nanochat with the IbnSina patches (training/nanochat_patches/, NANOCHAT_ARCH=qwen3) from github.com/ibnsina-llm.

path	what it is	notes
`base/big3b_190000/model_190000.pt` + `meta_190000.json`	final pretraining checkpoint (step 190,000; 99.6 B tokens; val bpb 0.4759)	the model every SFT below starts from
`base/big3b_190000/optim/optim_190000_rank{0..7}.pt`	Muon/AdamW optimizer state, 8 rank shards	for warm restarts
`base/tokenizer_TRAINING.tar.gz`	the exact tokenizer files used in training (`tokenizer.pkl` md5 7e262b94, `token_bytes.pt`)	the canonical tokenizer; do not mix with the pilot's `v1_32k`
`sft/v2_published/`	chat SFT v2 (`sft/v2` + MMLU auxiliary, 85 steps) — the model published on 2026-09-09 08:20 JST	evals in `results_big3b.json`
`sft/v21h_shipped/`	chat SFT v2.1 — the currently released weights (v2 data + 4,499 synthetic NLI/paraphrase/format rows + identity rows; 8×H100 run)	`results_big3b_v21h.json`, `diag_cls_big3b_v21h.json`
`sft/v21_seed2_unpublished/`	alternate SFT v2.1 run — NOT the release. Same data and recipe on one A100 (micro-batch 2); it did not reproduce the entailment gain (30.0 % vs 39.9 %). Kept as evidence that the v2.1 recipe is unstable	`results_big3b_v21.json`, `diag_cls_big3b_v21_seed2.json`
`release/ibnsina-3b/`, `release/ibnsina-3b-v21h/`, `release/ibnsina-3b-v21/`	the GGUF release bundles built from the three SFT checkpoints (v2 with F16 — do not use F16 on CUDA, it overflows; v2.1h = the public repo; v2.1 seed2 unpublished)
`MANIFEST.sha256`	sha256 of every file	verify after download

Evaluation numbers, the classification diagnostic and the v2.1 non-replication are documented on the model card and in the forthcoming family technical report. Licence: Apache-2.0 (weights and code). Author: Sina Meraji · ORCID 0009-0002-8028-1932.