shuff57/lfm2-24b-phase1-reasoning-GGUF

LFM2-24B-A2B Phase 1 Reasoning — GGUF Q4KM GGUF quantization of shuff57/lfm2-24b-phase1-reasoning (https://huggingface.co/shuff57/lfm2-24b-phase1-reasoning). Model Description This is the Phase 1 reasoning fine-tuned version of LiquidAI's LFM2-24B-A2B (24B MoE, 2.3B active parameters) model, trained on 13,201 synthetic reasoning examples generated for the O.G.R.E. (OllamaGradingRubricEvaluator) project. The base model (LiquidAI/LFM2-24B-A2B) was further fine-tuned using Unsloth's QLoRA approach with:

• —LoRA rank: 16
• —LoRA alpha: 16
• —Target modules: ["qproj", "kproj", "vproj", "outproj", "in_proj", "w1", "w2", "w3"]
• —Sequence length: 8192
• —Packing: False (critical for LFM2 MoE routing stability)
• —Training on responses only (using unsloth.chattemplates.trainonresponsesonly)
• —Optimizer: adamw_8bit
• —Learning rate: 2e-5
• —Epochs: 1 (effective steps: ~1,568)
• —Precision: bfloat16 After training, the LoRA adapters were merged into the base model and saved in 16-bit format. GGUF Quantization The merged model was converted to GGUF format for efficient CPU usage with llama.cpp and Ollama. Conversion Process
• —Model Loading: Loaded with device_map="cpu" to avoid MoE-related VRAM overflow during transformation.
• —HF → GGUF Conversion: Used the llama.cpp conversion pipeline:
• —Cloned llama.cpp (commit post-Oct 2025, includes native lfm2_moe support)
• —Configured with cmake -B llama.cpp/build -S llama.cpp -DCMAKEBUILDTYPE=Release
• —Built all targets (cmake --build llama.cpp/build --config Release)
• —Located the llama-quantize binary via glob search
• —Quantization: Applied Q4KM quantization: ./llama.cpp/build/bin/llama-quantize lfm2-24b-phase1-F16.gguf lfm2-24b-phase1-Q4KM.gguf Q4KM
• —Validation: The resulting GGUF file was tested with Ollama and llama.cpp for correct LFM2 MoE handling. Quantization Details
• —Format: GGUF (GPT-Generated Unified Format)
• —Quantization Method: Q4KM (4-bit with mixed precision, state-of-the-art balance)
• —Estimated Size: ~14.42 GB
• —VRAM Usage: Can be run on systems with <8GB VRAM when offloaded, or CPU-only
• —Context Length: 8192 tokens (matches training) Usage with Ollama Create a Modelfile: FROM ./lfm2-24b-phase1-Q4KM.gguf SYSTEM "You are a careful analytical reasoner. Think step by step before answering." RENDERER lfm2 PARSER lfm2 PARAMETER temperature 0.1 PARAMETER topk 50 PARAMETER topp 0.95 PARAMETER repeatpenalty 1.05 PARAMETER numctx 8192 Then: ollama create lfm2-24b-phase1-reasoning -f Modelfile ollama run lfm2-24b-phase1-reasoning "Your prompt here" Usage with llama.cpp llama-cli -m lfm2-24b-phase1-Q4KM.gguf -p "You are a careful analytical reasoner." --temp 0.1 -n 512 Training Data The Phase 1 reasoning model was trained on a synthetic dataset of 13,201 examples designed to teach step-by-step reasoning, logical deduction, and mathematical problem-solving. The dataset is available at:
• —shuff57/ogre-phase1-synth (https://huggingface.co/datasets/shuff57/ogre-phase1-synth) Related Models
• —Base Model: LiquidAI/LFM2-24B-A2B (https://huggingface.co/LiquidAI/LFM2-24B-A2B)
• —Phase 1 Reasoning (HF): shuff57/lfm2-24b-phase1-reasoning (https://huggingface.co/shuff57/lfm2-24b-phase1-reasoning) (16-bit merged weights)
• —Phase 2 Stat Grader: shuff57/lfm2-24b-grader (https://huggingface.co/shuff57/lfm2-24b-grader) (in progress) Citation If you use this model, please cite the O.G.R.E. project: @misc{ogre2026, title={O.G.R.E.: OllamaGradingRubricEvaluator}, author={shuff57}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/shuff57} } License Apache 2.0 — same as the base LFM2-24B-A2B model.