mps/blue-scrub-150M

blue-scrub-150M

Phase 5 Hugging Face deployment from slm-train-scratch.

Model summary

• —Model type: decoder-only causal language model (LlamaForCausalLM).
• —Model size: 150M-class; measured checkpoint size is 154.42M parameters.
• —Instruction tuning: Non-IT / non-instruction-tuned. This is a base language model, not a chat model and not instruction aligned.
• —Tokenizer: byte-level BPE-style tokenizer with vocabulary size 32000.
• —Precision: bfloat16 checkpoint.

Training data

The model was pretrained on a cleaned mixture of general educational text and medical/health-domain text.

• —General/education source: HuggingFaceFW/FineWeb-Edu, cleaned and filtered subset: 2.00M rows.
• —Medical/health source: TheBlueScrubs/thebluescrubs-v1, cleaned and filtered subset: 2.00M rows.
• —Packed train tokens: 5.72B tokens.
• —Packed validation tokens: 18.22M tokens.

Training run

• —Epochs configured: 1.
• —Approximate logged wall-clock training time: 73.3 hours (3.1 days). This is based on recorded metrics and may include evaluation, resume, and pause gaps.

Intended use

This checkpoint is intended for:

• —research experiments with small/base language models;
• —continued pretraining;
• —domain adaptation experiments;
• —evaluation of a compact medical-plus-education pretrained base model.

Limitations

• —This is not an instruction-tuned or chat-aligned model.
• —Outputs may be incomplete, incorrect, or unsafe without downstream alignment and evaluation.
• —The model must not be used as a source of medical advice.
• —The training mixture contains web and domain text; users should evaluate bias, factuality, memorization, and domain safety before downstream use.

Basic loading

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "mps/blue-scrub-150M"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype="auto")