Modelpublic

shreyash-pandey-katni/phoenix-350m-base

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes13downloads

Model Card

Phoenix-350M

A 350M-parameter decoder-only language model trained from scratch on ~10B tokens of FineWeb-Edu, on a single consumer GPU (RTX 3080 Ti, 12 GB). Released as a base model (no instruction tuning).

The architecture is a modern LLaMA/Qwen-style decoder, and the weights are provided in the stock Qwen3 format so they load with plain transformers (and export cleanly to ONNX / vLLM).

Highlights

—Chinchilla-optimal (~28.5 tokens/param) — trained to the compute-efficient point for its size.
—Punches above its token weight on knowledge/reasoning: on ARC it matches or beats models trained on 30×–180× more data, thanks to FineWeb-Edu's dense educational content.
—Final validation perplexity 15.6 (~0.86 bits/byte on its own held-out set).

Architecture


Type	Decoder-only (Qwen3-compatible)
Params	350M
Layers	24
d_model	1024
Heads	16 query / 4 KV (GQA)
Head dim	64, with QK-norm
FFN	SwiGLU, d_ff 3456
Norm	RMSNorm (pre-norm)
Positional	RoPE (θ=10000)
Context	2048
Vocab	32,000 (custom byte-level BPE)
Embeddings	tied

Training

—Data: FineWeb-Edu (sample-10BT), ~9.99B tokens, one epoch.
—Optimizer: 8-bit AdamW (bitsandbytes), lr 6e-4, WSD schedule (2000 warmup, final-10% linear decay).
—Precision: bf16, gradient checkpointing, global batch 524,288 tokens.
—Hardware: 1× RTX 3080 Ti (12 GB), ~7 days.

Evaluation (0-shot, `lm-evaluation-harness`, acc_norm where conventional)

Model	Train tok	HellaSwag	PIQA	ARC-e	ARC-c	WinoGrande	OBQA	SciQ
Phoenix-350M	10B	39.2	66.3	53.3	29.4	50.4	33.2	80.7
Pythia-410M	300B	33.7	66.8	52.1	21.3	53.7	30.0	—
OpenELM-450M (Apple)	~1.8T	54.0	72.3	48.1	27.6	58.0	—	87.2
OpenELM-270M (Apple)	~1.8T	46.7	69.8	45.1	26.5	53.9	—	84.7
MobiLlama-0.5B	~1.26T	52.5	72.0	—	29.5	57.5	—	—

Bits-per-byte (same-corpus, WikiText-103): Phoenix 0.98 vs Pythia-410M 0.86 vs Pythia-160M 1.19.

How to read this: Phoenix trained on 10B tokens — 30×–400× less data than the peers above. It's competitive-to-better on ARC (knowledge/reasoning, where FineWeb-Edu shines — it beats Apple's OpenELM-450M on both ARC-easy and ARC-challenge despite 180× less data), but its smaller token budget shows on HellaSwag, PIQA, WinoGrande and language-modeling metrics (BPB, LAMBADA), which scale with raw token count. MMLU is ~random at this scale, as expected for a 350M base model.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("shreyash-pandey-katni/phoenix-350m-base")
model = AutoModelForCausalLM.from_pretrained("shreyash-pandey-katni/phoenix-350m-base")
ids = tok("The process of photosynthesis begins when", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60)[0], skip_special_tokens=True))

ONNX weights are provided under `onnx/` for optimum / onnxruntime / transformers.js.

Limitations

—Base model — not instruction-tuned or aligned; it completes text, it doesn't follow chat instructions.
—English only, ~10B tokens — limited world knowledge; will hallucinate and repeat.
—No safety/RLHF filtering. Use for research and as a small-model baseline, not production.

License

Apache-2.0. Trained on FineWeb-Edu (ODC-BY).