ryze-ai/flash-archer-150M-2.0

Flash-Archer-150M-2.0

A clean, decoder-only transformer language model trained from scratch on FineWeb-Edu — 150M parameters, ~2.1B tokens, on a single Tesla T4 in ~7 hours. Built as a self-contained, reproducible pretraining project: custom byte-level BPE tokenizer, streaming data pipeline, cosine LR schedule with warmup, and a hand-written PyTorch training loop.

This is the v2.0 release: the same architecture and training recipe, packaged as a standard HuggingFace LlamaForCausalLM so it works everywhere out of the box — transformers, vLLM, TGI, llama.cpp, pipelines — with bit-identical inference to the original training/inference scripts (verified: logit max-diff < 1e-5).

Model details

Quickstart

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Norman89107/Flash-Archer-150M-2.0"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()

prompt = "Photosynthesis is the process by which"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=120,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Sampling tips

• —Lower temperature (0.3–0.6) → more focused, factual output
• —Higher temperature (0.9–1.2) → more creative, more repetition
• —top_k=0 disables top-k; top_p=1.0 disables nucleus sampling
• —Context is 1024 tokens; longer prompts are cropped from the left

Training recipe

The full training code, tokenizer training, and streaming data pipeline are in the accompanying notebooks (archer-150m.ipynb for training, archer-150m-inference-colab.ipynb for inference). Key design choices:

• —Streaming, never materialized — FineWeb-Edu is streamed token-by-token; a fixed 2,000-doc holdout is reserved for validation and never seen by training.
• —Packed sequences — documents are concatenated with <eos> separators into fixed 1024-token chunks, so there is zero padding waste.
• —Activation checkpointing + micro-batch size 1 + gradient accumulation 128 — keeps the effective batch at ~131k tokens/step while fitting 1024-token training in 16 GB VRAM.
• —Persistent checkpoints — every 200 steps the model + optimizer + logs are mirrored to the HuggingFace Hub, so a crashed Kaggle session loses at most ~200 steps.

Intended use & limitations

• —Intended use: a compact, fast, English language model for experimentation, education, and as a baseline for small-model research. It writes coherent, on-topic prose and follows simple instructions.
• —Limitations: at 150M parameters it will hallucinate facts, repeat itself (especially at high temperature), and struggle with multi-step reasoning, math, and code. It has no safety alignment or RLHF — do not use it for applications without additional safeguards. Trained on web data; it may reflect biases and content present in FineWeb-Edu.

Conversion & reproducibility

This release was converted from the original training checkpoint (ckpt_step0016200.pt) into standard HuggingFace LlamaForCausalLM format. The only architectural difference is the RoPE convention (interleaved → half-split), handled by a fixed Q/K head-dimension permutation during conversion. Inference parity was verified: the converted model's logits match the original model's to within float32 rounding (max |diff| < 1e-5), and seeded generation produces identical output. See convert_to_hf.py and verify_parity.py in the project source.

Citation

If you use this model, please cite the training data and this repository:

@misc{flash-archer-150m-2,
  title  = {Flash-Archer-150M-2.0},
  author = {Norman89107},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Norman89107/Flash-Archer-150M-2.0}},
  note   = {Decoder-only transformer trained from scratch on FineWeb-Edu (sample-10BT), ~2.1B tokens.}
}

@dataset{fineweb-edu,
  title  = {FineWeb-Edu},
  author = {Hugging Face},
  url    = {https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu}
}