yasserrmd/LFM2.5-1.2B-onpolicy

LFM2.5-1.2B-onpolicy

A high-performance 1.2B parameter language model fine-tuned from LFM2.5-1.2B-Instruct using on-policy self-distillation, inspired by Self-Distillation Fine-Tuning (Shenfeld et al., 2026) but with simplified demonstration handling. Optimized for instruction-following and general-purpose assistant tasks.

Model Overview

Training Details

Fine-Tuning Approach: On-Policy Self-Distillation

This implementation is inspired by SDFT (Shenfeld et al., 2026) but uses a simplified approach to on-policy learning:

✅ SDFT Features Implemented

1. 1.On-Policy Learning:
2. 2.Student model generates trajectories: y ~ π_θ(·|x)
3. 3.Both student and teacher evaluated on same generated sequences
4. 4.Prevents off-policy distribution shift

1. 1.Reverse KL Divergence:
2. 2.Minimizes: D_KL(π_θ ∥ π)
3. 3.Computed token-level via log probability differences
4. 4.Prevents mode collapse

1. 1.Teacher as Base Model:
2. 2.Teacher = LFM2.5-1.2B-Instruct (same architecture)
3. 3.Self-distillation from stronger base model
4. 4.Preserves pre-training knowledge

1. 1.EMA Teacher Updates:
2. 2.Teacher parameters exponentially averaged from student
3. 3.Stable target for KL divergence
4. 4.EMA coefficient: α = 0.02 (slow decay)

1. 1.8-Bit Quantization:
2. 2.Both models in 8-bit via BitsAndBytes
3. 3.Memory efficient (1.2GB per model)
4. 4.Reduces GPU memory footprint to 35GB peak

❌ SDFT Features Omitted (Simplified)

1. 1.Demonstration Conditioning ⚠️
2. 2.Paper Definition: Teacher conditioned on both query and demonstration π(·|x, c)
3. 3.This Implementation: Teacher only sees student-generated tokens π(·|x)
4. 4.Rationale: Simplified to focus on on-policy distillation without explicit demonstration signals
5. 5.Impact: Loss of implicit reward signal from demonstrations

1. 1.Demo-Aware Logits in Loss ⚠️
2. 2.Paper: Loss computed on π_θ(y|x) vs π(y|x,c) divergence
3. 3.This Implementation: Loss computed on π_θ(y|x) vs π(y|x) divergence
4. 4.Impact: Teacher provides base model alignment, not demo-based guidance

1. 1.Analytic Per-Vocabulary KL ⚠️
2. 2.Paper: Full vocabulary KL over all tokens (lower variance): Σ_y π_θ(y) log(π_θ/π)
3. 3.This Implementation: Sampled-token KL only on generated tokens (higher variance): log π_θ(y_t) - log π(y_t)
4. 4.Paper Note: Explicitly cautions this estimator is "higher variance and less stable"
5. 5.Impact: Higher gradient variance, potentially slower convergence

1. 1.Explicit Demonstration Prompt Template ⚠️
2. 2.Paper: Teacher uses explicit in-context prompt: <Query> This is an example: <Demo> Now answer...
3. 3.This Implementation: Demonstrations prepared in dataset but not applied during teacher forward pass
4. 4.Impact: Demonstrations unused during training despite preparation

Key Design Rationale

This simplified approach trades off the full SDFT mechanism for:

• —Simplicity: Single forward pass per model (not demo-conditioned)
• —Efficiency: Reduced computational overhead during training
• —Focus: Concentrates on on-policy learning benefits without demo conditioning complexity
• —General Fine-Tuning: Better suited for instruction tuning than continual learning from expert demos

Dataset

Note: Demonstrations are prepared and tokenized in dataset but not used in training loop (simplified approach).

Training Configuration

Hardware and Infrastructure

Training Environment:

• —GPU: NVIDIA A100 80GB
• —System RAM: 167.1 GB (used 8.4 GB during training)
• —Disk Storage: 235.7 GB (used 41.2 GB)
• —Framework: PyTorch 2.x + Transformers + peft
• —Quantization: BitsAndBytes (8-bit for both student and teacher models)

Memory Usage:

• —Student model (8-bit): 1.2 GB
• —Teacher model (8-bit): 1.2 GB
• —Batch tensors: 4 GB
• —Activations and Gradients: 3-5 GB
• —Total Peak: 35 GB (44% of 80GB available)

Performance Evaluation

Evaluation Results (90 test prompts)

Performance by Category

Key Findings:

• —Strong performance on knowledge, technical, and problem-solving tasks
• —Comprehensive responses with proper length calibration
• —Preserved base model knowledge through on-policy distillation
• —Consistent output across diverse domains

Usage

Installation

pip install transformers torch

Quick Start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "yasserrmd/lfm2.5-1.2b-onpolicy",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("yasserrmd/lfm2.5-1.2b-onpolicy")

# Generate response
prompt = """<|im_start|>user
What is machine learning?
<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.95
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Generation Parameters

Model Cards and Licenses

• —Base Model License: Check LiquidAI/LFM2.5-1.2B-Instruct
• —Training Data License: OASST1 (Creative Commons)
• —Fine-tuned Model: Open for research and commercial use (check base model license)

Technical Insights

Why On-Policy Self-Distillation Works

1. 1.Knowledge Preservation: Base model alignment prevents catastrophic forgetting
2. 2.Efficient Learning: 8-bit quantization reduces memory overhead
3. 3.Stability: KL divergence from base model provides stable training signal
4. 4.Practical: Runs efficiently on consumer GPUs with parameter-efficient training

Differences from Paper's SDFT

This implementation follows the spirit of SDFT (on-policy learning) but uses a simplified approach:

Result: On-policy self-distillation optimized for instruction tuning rather than demonstration-based continual learning.

Training Observations

• —Loss Trajectory: Started at 0.1460, converged to 0.1845 at epoch completion
• —Memory Efficiency: Scaled to batch_size=16 with 35GB peak usage (44% utilization)
• —Convergence: Stable training without divergence or OOM errors
• —Single Epoch: Sufficient quality for production deployment without requiring multiple epochs

Deployment

The model is ready for:

• —Chat and conversational AI applications
• —Question-answering systems
• —Content generation (summaries, explanations)
• —Instruction-following tasks
• —Fine-tuning for domain-specific applications

Recommended Hardware

Citation

If you use this model in your research, please cite:

@misc{lfm25_onpolicy_2026,
  author = {Yasser RMD},
  title = {LFM2.5-1.2B: On-Policy Self-Distillation Fine-Tuned Language Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/yasserrmd/lfm2.5-1.2b-onpolicy}},
  note = {Inspired by Shenfeld et al. (2026) SDFT}
}

References

• —Self-Distillation Fine-Tuning (SDFT): Shenfeld, I., Damani, M., Hübotter, J., & Agrawal, P. (2026). "Self-Distillation Enables Continual Learning." arXiv preprint arXiv:2601.19897. https://arxiv.org/abs/2601.19897
• —Inspiration: This work inspired the on-policy distillation approach
• —Key Difference: This implementation omits demonstration conditioning for simplified instruction tuning

• —Base Model: LiquidAI LFM2.5-1.2B. https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct

• —Dataset: OpenAssistant/OASST1. https://huggingface.co/datasets/OpenAssistant/oasst1

• —Training Framework: Hugging Face Transformers & PEFT (Parameter-Efficient Fine-Tuning). https://github.com/huggingface/peft

Limitations

1. 1.Simplified Approach: Does not implement full SDFT with demonstration conditioning (see Technical Insights section)
2. 2.Single Epoch: Trained for 1 epoch only; potential improvements possible with additional epochs
3. 3.Dataset Size: 5,000 samples is relatively small; larger datasets could improve generalization
4. 4.Domain Bias: Trained on general assistant tasks; may underperform on specialized domains
5. 5.Knowledge Cutoff: Inherited from base model pre-training
6. 6.Factual Accuracy: Q&A on recent events may be less accurate than larger models
7. 7.KL Estimator: Uses sampled-token KL (higher variance) instead of analytic vocabulary KL from paper

Contributing

For improvements, issues, or collaborations:

• —Open an issue on the Hugging Face Hub
• —Submit pull requests with improvements
• —Share evaluation results or use cases

License

This model follows the license of the base model (LFM2.5-1.2B-Instruct). Please refer to the original model repository for details.

Created: February 2026 Training Hardware: NVIDIA A100 80GB Status: Production Ready (Single Epoch, On-Policy Self-Distilled) Inspiration: SDFT (Shenfeld et al., 2026) Implementation Type: Simplified Instruction Fine-Tuning Variant