CoolFace
Modelpublic

JulianKrgd/julian-600m-40b-instruct-sft100k

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face
0likes327downloads
Model Card

Julian 600M-40B Instruct SFT-100K

Fine-tuned version of the Julian 600M base model with 100,000 steps of supervised fine-tuning (SFT) on instruction-following data.

Model Details

ParameterValue
ArchitectureLlamaForCausalLM (decoder-only)
Parameters~600M
Hidden size1280
Layers18
Attention heads20
FFN size5120 (SwiGLU)
Vocab size50,000 (SentencePiece)
Context length2048
Precisionbfloat16
NormRMSNorm
Position encodingRoPE

Benchmark Results (0-shot)

Julian Models

BenchmarkBase (39B tokens)SFT 30KSFT 100K
HellaSwag (acc_norm)53.5%41.7%41.6%
PIQA (acc)66.8%66.8%66.6%
LAMBADA (acc)37.3%37.7%37.7%
LAMBADA (ppl↓)—15.3815.33
ARC Easy (acc)—53.5%53.8%
ARC Challenge (acc_norm)—27.1%26.7%
WinoGrande (acc)—53.8%52.8%
BoolQ (acc)—60.6%60.8%
SFT 30K and 100K yield near-identical benchmark scores. Additional SFT steps beyond 30K don't improve knowledge benchmarks. WinoGrande starts to degrade at 100K (likely overfitting on ~2.2 epochs).

Comparison with Other Models

ModelParamsTokensHellaSwagPIQALAMBADAARC-EARC-CWinoGrande
GPT-2 Small124M100B+31.5%—46.0%——50.4%
OPT-125M125M300B29.2%63.0%37.9%43.5%18.9%50.3%
OPT-350M331M300B32.0%64.4%45.2%44.0%20.7%52.3%
Pythia-410M405M300B33.3%66.8%50.5%50.4%21.3%53.0%
Julian 600M SFT-100K600M39B+6.5B41.6%66.6%37.7%53.8%26.7%52.8%
Julian 600M Base600M39B53.5%66.8%37.3%———
GPT-2 XL1558M100B+50.9%70.8%63.2%——59.4%
Pythia-1B1B300B37.6%70.5%56.6%55.9%24.3%54.5%
OPT-1.3B1.3B300B41.5%71.7%57.9%57.0%23.4%59.5%
Julian 600M Base outperforms OPT-1.3B on HellaSwag (53.5% vs 41.5%) despite being 2x smaller and trained on 8x fewer tokens. The SFT version trades some HellaSwag performance for instruction-following ability, while maintaining competitive scores on PIQA, ARC, and WinoGrande. Sources: GPT-2 — OpenAI; OPT — Meta; Pythia — EleutherAI

Training

Base Model

  • —Pre-training: ~40B tokens (70% EN / 30% FR)
  • —Data: Wikipedia, OSCAR, Gutenberg, The Stack
  • —Infrastructure: TPU v4-32, JAX/Flax

SFT Fine-tuning

  • —Steps: 100,000 (from pretrained checkpoint_300000)
  • —Dataset: 2.47M instruction examples (tokenized)
  • —Batch size: 32 global (2/device × 4 devices × 4 hosts)
  • —Sequence length: 2048
  • —Epochs: ~2.2
  • —Tokens seen: ~6.55B
  • —Final loss: 1.69
  • —Infrastructure: TPU v4-32, JAX/Flax

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("JulianKrgd/julian-600m-40b-instruct-sft100k", torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("JulianKrgd/julian-600m-40b-instruct-sft100k")

inputs = tokenizer("Explain quantum computing in simple terms:", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

  • —Small model (600M params) — limited reasoning and factual accuracy
  • —Instruction following is basic compared to larger models
  • —May hallucinate or generate incorrect information
  • —Bilingual (EN/FR) but stronger in English
  • —2.2 epochs over training data — some overfitting possible

Framework

Trained from scratch using JAX/Flax on Google Cloud TPU v4-32. Converted to HuggingFace safetensors format for compatibility.

License

Apache 2.0