safal312/Iterative-Qwen-7layers-SGR-Dolci

Iterative-Qwen-7layers-SGR-Dolci

This model is a depth-pruned version of Qwen2.5-7B-Instruct, obtained via Iterative layer pruning and post-trained using QLoRA with Self-Generated Responses (SGR) on the Dolci dataset.

It was released as part of our study on the limits of layer pruning for generative reasoning.

Summary

• —Base model: Qwen2.5-7B-Instruct
• —Pruning: Iterative
• —Depth: 7 layers
• —Post-training: QLoRA
• —Supervision: Self-Generated Responses (SGR)
• —Data: Dolci (SGR variant)

This checkpoint is intended for research and analysis of pruning and recovery, not as a production model.

Paper

On the Limits of Layer Pruning for Generative Reasoning in LLMs https://arxiv.org/abs/2602.01997

@misc{shrestha2026limitslayerpruninggenerative,
  title={On the Limits of Layer Pruning for Generative Reasoning in LLMs},
  author={Safal Shrestha and Anubhav Shrestha and Aadim Nepal and Minwu Kim and Keith Ross},
  year={2026},
  eprint={2602.01997},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2602.01997},
}