Omni-Post-Train/LFM2.5-Audio-1.5B-IFEval-SFT-GRPO

LFM2.5-Audio IFEval — evaluated SFT/GRPO continuation

Main contains the exact unmerged adapter selected as sft-cycle2-a, SHA256 8dfe0deba6e38848795fb21457a5613c939503bea8693e9ecb0759f0bb54b37d. Both new SFT experiments are archived under checkpoints/cycle2-a and checkpoints/cycle2-b; their presence does not imply that they passed promotion guards. The original SFT + GRPO release is retained at tag cycle1.

Every score uses 345 prompts, 498 checks, 24 families, audio input and text responses. A fixed per-question random seed makes the upstream question-1122 # fallback reproducible; it does not fix that defect. Historical cycle1 scores remain preserved. This continuation is benchmark-informed. The new held-out synthetic test and native speech results are in the full report.

Training used one A100 40GB. Stage A used 747 tasks and continued the published SFT + GRPO adapter for one SFT epoch at 2e-5; stage B continued A with 1011 tasks for one epoch at 1.5e-5. Effective batch size was 16. Only the 11,108,352 backbone LoRA parameters were trainable. The existing GRPO history remains 12 prompts, 48 native rollouts and 24 updates with Whisper large-v3; this round did not add GRPO updates.

Selection ranked full new-development IFEval while guarding content quality, native instruction compliance, native WER, transcript availability and unfinished text responses. The published initializer remained eligible, so a text gain alone could not force replacement. See cycle2_selection.json, each stage's complete analysis in reports/, and release_manifest.json for weights and evidence hashes. Other VoiceBench metrics, human perceptual quality, and multiple-seed stability remain untested.

Each development evaluation contains all 166 prompts. Native WER is measured over 160 transcribed outputs per checkpoint; six tasks per checkpoint have no transcript and remain in the other evaluation denominators. The initializer's native WER is 0.0950; the frozen maximum allowed increase is 0.02. The selected adapter uses dataset configuration cycle2-a. Passing a tolerance does not establish better speech quality, and the official score-change confidence interval crosses zero.

The reserved test was opened after selection and contains 158 prompts in each mode:

The reserved text-score improvement is inconclusive, native instruction compliance declines slightly, and the joint native success count decreases. This is an experimental continuation with mixed results, not an established speech-to-speech improvement. Selection was not changed after observing this test.

The reviewed findings document concrete repetition, formatting and factual errors observed in these experiments. Mechanical instruction success is not a factual-quality certification. Official evaluation prompts and contracts retain their upstream Apache-2.0 attribution under evaluation/.

Inference with the required Transformers branch

pip install huggingface_hub
hf download Omni-Post-Train/LFM2.5-Audio-1.5B-IFEval-SFT-GRPO --local-dir checkpoint
pip install -r checkpoint/requirements.txt
pip install peft==0.20.0
python checkpoint/inference.py --adapter checkpoint --audio question.wav --max-new-tokens 4096 --output-json answer.json

The requirements pin kadirnar/transformers commit 843101f38c800b98d49b704215f0b76b92e48e64 on add-lfm2-audio. Alternatively attach the downloaded adapter with model.model.lfm = PeftModel.from_pretrained(model.model.lfm, adapter_directory) after loading the complete native audio checkpoint. PeftModel.from_pretrained on the top-level audio model is not the intended attachment point. See lfm_audio_adapter.json for exact model and weight identities.

The base model and derivatives are subject to the included LFM Open License v1.0. This is a research release trained with synthetic audio from Echo/Fish components having noncommercial terms; the dataset is CC BY-NC-SA 4.0. See their linked licenses and NOTICE.md. No upstream endorsement is implied.