pyromind/PyroDash-4B-GRPO-Lambda-0.6

PyroDash-4B-GRPO-Lambda-0.6

This repository hosts PyroDash-4B-GRPO-Lambda-0.6 — the GRPO Stage 3 checkpoint with efficiency penalty λ = 0.6 (cost-oriented), fine-tuned from Qwen/Qwen3.5-4B via PyroDash-4B-SFT.

Companion models: SFT · GRPO λ=0.05

<table> <tr> <td width="50%" valign="top"> <img src="https://raw.githubusercontent.com/PyroMind-Dynamics/pyroDash/main/docs/assets/inference.png" alt="Inference Architecture" width="100%"/> </td> <td width="50%" valign="middle">

We propose PyroDash, a token-level dynamic reasoning paradigm for collaborative inference between small and large language models. PyroDash enables the small model to autonomously emit the control token <|llm_offload|> during autoregressive streaming decoding; the collaboration engine then dynamically offloads the local reasoning chain to a large model based on this control signal. This approach requires neither an additional router model nor retraining of the large model, and is naturally compatible with closed-source LLM services.

</td> </tr> </table>

During training, PyroDash follows a three-stage progressive optimization pipeline: (1) train the control-token embedding layer so the small model acquires basic offloading expressiveness; (2) cold-start the offload capability to establish a collaboration pattern between the small and large models; and (3) apply GRPO (this checkpoint, λ=0.6) that jointly optimizes the dynamic offloading policy with a task-accuracy reward and a large-model call-cost penalty, achieving an adaptive balance between reasoning quality and compute cost.

<p align="center"> <img src="https://raw.githubusercontent.com/PyroMind-Dynamics/pyroDash/main/docs/assets/training.png" alt="Three-stage progressive training pipeline" width="80%"/> </p>

Model Details

Quick Start

1. Setup

git clone https://github.com/PyroMind-Dynamics/pyroDash.git
cd pyroDash
pip install -r requirements.txt

2. Run evaluation (evaluation/math_eval.sh)

Edit placeholders in `evaluation/math_eval.sh`, then:

bash evaluation/math_eval.sh

The script (1) starts a local vLLM server for the small model on port 8001, (2) runs math_eval.py, and (3) stops vLLM on exit.

Parameters

Tokenizer must include the special token <|llm_offload|>.

Results

<p align="center"> <img src="https://raw.githubusercontent.com/PyroMind-Dynamics/pyroDash/main/docs/assets/figcostaccuracy_pareto.png" alt="Cost–Accuracy Pareto" width="70%"/> </p>

λ=0.6 is the cost operating point (~1.9% LLM tokens; ~$1.78 vs $49.36 for GLM-only). For peak accuracy, use λ=0.05.

Resources

Citation

@misc{pyrodash2026,
  title        = {PyroDash: Cost-Efficient Token-Level Small-Large Model Collaborative Inference},
  author       = {{PyroMind Dynamics}},
  year         = {2026},
  note         = {Preprint}
}

@misc{pyromind2026easyhard24k,
  title        = {{EasyHard-24K} v0.02},
  author       = {{PyroMind Dynamics}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/pyromind/easyhard-24k}}
}

GRPO training data: BytedTsinghua-SIA/DAPO-Math-17k.

License

Apache 2.0 (derived from Qwen3.5-4B).