alea-institute/kl3m-007-500m-step100000

KL3M 500M Conservative, Step 100000

A 500M parameter language model trained with conservative hyperparameters using the Muon optimizer and spectral clamping. This checkpoint represents 100,000 training steps (8.32B tokens).

Model Details

• —Architecture: Llama-based with Grouped Query Attention (GQA)
• —Parameters: 500.3M
• —Layers: 120
• —Training Steps: 100,000
• —Tokens Processed: 8.32B
• —Precision: BF16

Training Configuration

Optimizer: Muon

• —Muon LR: 0.000073 (depth-scaled)
• —Aux LR: 0.00005
• —Momentum: 0.95

Spectral Clamping

• —Frequency: Every 100 steps
• —Max condition: 2500
• —Applied to: attention, MLP, and lm_head layers

Regularization

• —Label Smoothing: 0.01
• —Entropy Bonus: 0.005
• —Activation Norm: 0.0005

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "alea-institute/kl3m-007-500m-step100000",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("alea-institute/kl3m-007-500m-step100000")

inputs = tokenizer("The contract specifies", return_tensors="pt", return_token_type_ids=False)
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.8, top_p=0.95)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

Apache 2.0