alea-institute/kl3m-006-170m-checkpoint-33000

KL3M 170M, 6th Gen Model, 33K Checkpoint

A 170M parameter language model trained on legal agreements using the Muon optimizer with spectral clamping.

Model Details

• —Architecture: Llama-based with Grouped Query Attention (GQA)
• —Parameters: 181.7M (170M non-embedding)
• —Training Steps: 33,000
• —Sequence Length: 4,096 tokens
• —Precision: BF16
• —Optimizer: Muon with spectral regularization (max condition: 2000)

Model Architecture

• —Hidden Size: 576
• —Layers: 30
• —Attention Heads: 9 (3 KV heads with GQA)
• —Intermediate Size: 1536
• —Vocabulary: 131,072 tokens

Training Configuration

• —Dataset: Legal agreements (EDGAR filings)
• —Optimizer: Muon with momentum 0.95
• —Muon Learning Rate: 8e-5 (depth-scaled)
• —Auxiliary Learning Rate: 4e-5
• —Batch Size: 1 per device (effective 4 with gradient accumulation)
• —Gradient Accumulation Steps: 4
• —Warmup Steps: 10,000
• —LR Scheduler: Cosine with warmup
• —Weight Decay: Muon 1e-5, Auxiliary 0.001
• —Spectral Clamping: Enabled (max condition 2000, sigma floor 6e-4, every 10 steps)
• —Mixed Precision: BF16
• —Gradient Checkpointing: Enabled
• —Additional Regularization:
• —Entropy bonus weight: 0.001 (target: 6.5 bits)
• —Activation norm weight: 0.001
• —Loss chunk tokens: 1024

Spectral Health (Step 33K)

• —Attention Layers: Median condition number 237.75 ✓ EXCELLENT
• —MLP Layers: Median condition number 4.58 ✓ EXCELLENT
• —Max Attention Condition: 2208.16 (at spectral clamp ceiling)

Generation Quality

Generates coherent, fluent legal text with no repetition issues. Best for legal/contractual content.

Usage

from transformers import pipeline

# Create text generation pipeline
generator = pipeline(
    "text-generation",
    model="alea-institute/kl3m-006-170m-checkpoint-33000",
    torch_dtype="auto",
    device_map="auto"
)

# Generate text
outputs = generator(
    "This Agreement is entered into as of",
    max_new_tokens=100,
    temperature=0.7,
    top_p=0.95,
    repetition_penalty=1.15
)

print(outputs[0]['generated_text'])

Citation

For technical details, see the paper: https://arxiv.org/abs/2504.07854

@misc{kl3m2025,
  title={KL3M: Knowledge-Guided Language Model Training},
  author={Alea Institute},
  year={2025},
  url={https://arxiv.org/abs/2504.07854},
  note={Trained with Muon optimizer and spectral clamping}
}

License

Apache 2.0