JulianKrgd/JULIAN-100M-Instruct

JULIAN-100M-Instruct

Instruction-tuned version of JULIAN-100M - A 109M parameter bilingual (English/French) language model fine-tuned for instruction following.

Model Description

JULIAN-100M-Instruct is the instruction-tuned version of JULIAN-100M, fine-tuned on ~143K instruction-response pairs in English and French.

Training Data

Fine-tuned on 142,812 instruction-response pairs:

Usage

With JAX/Flax

import jax.numpy as jnp
from huggingface_hub import hf_hub_download
import sentencepiece as spm
import numpy as np

# Load tokenizer
tokenizer_path = hf_hub_download(
    repo_id="JulianKrgd/JULIAN-100M-Instruct",
    filename="tokenizer/julian_24k.model"
)
tokenizer = spm.SentencePieceProcessor()
tokenizer.Load(tokenizer_path)

# Load checkpoint
checkpoint_path = hf_hub_download(
    repo_id="JulianKrgd/JULIAN-100M-Instruct",
    filename="checkpoint_instruct.npz"
)

# Format prompt (ChatML)
prompt = """<|im_start|>user
What is machine learning?<|im_end|>
<|im_start|>assistant
"""

# Tokenize and generate...

Prompt Format

The model uses ChatML format:

<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

Limitations

This is a small educational model with significant limitations:

• —Factual errors: May generate incorrect information (e.g., "Paris is in New York")
• —Repetitions: Tends to repeat phrases in loops
• —Limited knowledge: 100M parameters cannot store much factual knowledge
• —Hallucinations: Confidently generates false statements
• —Not for production: This is a demonstration model, not a reliable assistant

Recommended minimum for usable responses: 7B+ parameters

Training Details

• —Base model: JULIAN-100M (trained on 4.45B Wikipedia tokens)
• —Fine-tuning: Supervised Fine-Tuning (SFT) with loss masking
• —Loss masking: Only computed loss on assistant responses
• —Learning rate: 2e-5 with warmup
• —Batch size: 128 global
• —Hardware: Google Cloud TPU v5e-32 (32 cores)
• —Training time: ~17 minutes

Model Architecture

- Vocabulary: 24,000 tokens (SentencePiece BPE)
- Embedding dim: 640
- Layers: 12
- Attention heads: 10
- MLP dim: 2560 (4x)
- Techniques: RMSNorm, RoPE, SwiGLU, bfloat16

Files

• —checkpoint_instruct.npz - Model weights (NPZ format)
• —tokenizer/julian_24k.model - SentencePiece tokenizer
• —src/model/ - Model code (JAX/Flax)

Citation

@misc{julian-100m-instruct,
  author = {Julian Kerignard},
  title = {JULIAN-100M-Instruct: A Small Bilingual Instruction-Tuned Language Model},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/JulianKrgd/JULIAN-100M-Instruct}
}

Acknowledgments

• —Trained with Google TPU Research Cloud (TRC) program
• —Based on instruction datasets from the open-source community

License

MIT License