nasos10/MuToR-gemma-2B-GSM8K-dmax_4_a_03

MuToR: Multi-Token prediction with Registers

Arxiv: https://arxiv.org/abs/2505.10518

TL;DR: MuToR is a simple, plug-and-play approach for multi-token prediction. It leverages dummy register tokens to predict multiple targets in the future, enriching the supervisory signal and improving performance across diverse settings and modalities. The register tokens are discarded on inference, leaving generation speed unchanged.

Model Description

This model is a finetuned version of Gemma-2B. It was finetuned using the MuToR method for 5 epochs on the GSM8K training split. Please refer to our code for guidelines on how to use the models to reproduce our results.