ermiaazarkhalili/LFM2-1.2B-GRPO-NuminaMath-10K

LFM2-1.2B-GRPO-NuminaMath-10K

This model is a fine-tuned version of LiquidAI/LFM2-1.2B trained on the AI-MO/NuminaMath-CoT dataset using Group Relative Policy Optimization (GRPO) - an online reinforcement learning method.

Overview

LFM2-1.2B-GRPO-NuminaMath-10K is optimized for mathematical reasoning tasks. It uses GRPO to learn from reward signals based on answer correctness and format adherence, enabling it to generate more accurate step-by-step solutions.

Key Features

• —Reinforcement Learning: Trained with GRPO for improved reasoning capabilities
• —Math Focus: Optimized on 10,000 math problems from NuminaMath-CoT
• —Multi-Sample Learning: Uses 2 generations per prompt for robust training
• —Combined Reward: Evaluates both answer accuracy and output format

Model Details

Training Information

GRPO Configuration

LoRA Configuration

Training Metrics

Reward Function

The model uses a combined reward function:

1. 1.Math Accuracy: Extracts and validates final numerical answers
2. 2.Format Compliance: Checks for proper step-by-step reasoning format
3. 3.Combined Score: Weighted combination of accuracy and format rewards

Training Hardware

• —GPU: NVIDIA H100 40GB MIG (3g.40gb)
• —CPU: 8 vCPUs
• —Memory: 64GB
• —Platform: Compute Canada (Fir Cluster)

Dataset

This model was trained on the AI-MO/NuminaMath-CoT dataset:

NuminaMath-CoT provides step-by-step mathematical solutions, enabling the model to learn structured reasoning patterns.

Usage

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ermiaazarkhalili/LFM2-1.2B-GRPO-NuminaMath-10K"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Math problem
prompt = "Solve step by step: If a train travels 120 km in 2 hours, what is its average speed?"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Limitations

• —Domain Specific: Optimized for math; may not generalize to other reasoning tasks
• —Language: English only
• —Hallucinations: May produce incorrect calculations despite correct format
• —Verification Needed: Always verify mathematical results independently

Intended Use

Recommended Uses

• —Mathematical problem solving
• —Step-by-step reasoning demonstrations
• —Educational math tutoring applications
• —Research on RL-trained language models

Out-of-Scope Uses

• —Critical calculations requiring absolute accuracy
• —Non-mathematical reasoning tasks
• —Production systems without verification

Citation

@misc{ermiaazarkhalili_lfm2_1.2b_grpo_numinamath_10k,
    author = {Ermia Azarkhalili},
    title = {LFM2-1.2B-GRPO-NuminaMath-10K: GRPO-trained LFM2-1.2B for Math},
    year = {2025},
    publisher = {Hugging Face},
    howpublished = {\url{https://huggingface.co/ermiaazarkhalili/LFM2-1.2B-GRPO-NuminaMath-10K}}
}

Acknowledgments

• —LiquidAI for the LFM2 base model
• —Hugging Face TRL Team for the GRPO implementation
• —NuminaMath dataset creators
• —Compute Canada for providing HPC resources