Modelpublic

MikhailRudenko/drafter-wmt16-translate-ruen

sourceHugging Faceapache-2.0updated 13d agoView on Hugging Face

0likes162downloads

Model Card

Drafter — WMT16 Translate RUEN (Russian → English)

A domain-specific draft model for speculative decoding, fine-tuned on the wmt16_translate_ruen cluster from the Flan dataset.

Model Details

Parameter	Value
Architecture	MistralForCausalLM (156M parameters)
Base model	Lite-Mistral-150M-v2-Instruct
Target model	TurboSparse-Mistral-Instruct (7B, BambooForCausalLM)
Domain	WMT16 Translation: Russian → English
Training cluster	`wmt16_translate_ruen_10templates`
Precision	bfloat16

Training

Parameter	Value
Loss function	0.5 × CrossEntropy + 0.5 × KL-divergence (T=1.0)
Training data	28,500 synthetic samples (knowledge distillation from target top-10 distributions)
Validation data	1,500 samples (5% held-out split)
Epochs	10 (plateau at ~5)
Batch size	32
Learning rate	5e-5 (cosine schedule, 3% warmup)
Best checkpoint	`checkpoint-7565`
Final eval_loss	2.363
Final top1_accuracy	50.04%
GPU	1× RTX 3090 (24 GB)

Purpose

This model is part of the Domain-Aware Speculative Decoding research project. The hypothesis is that domain-specific draft models achieve higher acceptance rates in speculative decoding compared to a single general-purpose drafter.

The model is trained via knowledge distillation: the target model's top-10 token distributions are used as soft labels, combined with standard cross-entropy on the greedy (trunk) sequence.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-wmt16-translate-ruen")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-wmt16-translate-ruen")

Related Models

Model	Domain	eval_loss	top1_acc
drafter-understanding	Understanding (31 clusters)	2.100	65.00%
drafter-text-reformulation	Text Reformulation (11 clusters)	2.151	54.34%
drafter-mixed-ut	Mixed U+T (42 clusters)	2.085	59.50%
drafter-wmt16-translate-tren	Turkish → English	1.998	54.91%
drafter-wmt16-translate-deen	German → English	2.307	52.77%
drafter-wmt16-translate-ruen	Russian → English	2.363	50.04%
drafter-wmt16-translate-csen	Czech → English	2.260	49.96%
drafter-wmt16-translate-fien	Finnish → English	2.252	49.36%

Dataset

Training data: MikhailRudenko/domain-aware-sd-synthetic