CoolFace
Modelpublic

kmouratidis/Devstral-Small-2507-Rebased-Vision

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face
0likes11downloads
Model Card

Devstral-Small-2507-Rebased-Vision

This model was created by taking Mistral-Small-3.2-24B-Instruct-2506 and replacing the weights under the language_model with the weights from Devstral-Small-2507. The result is Devstral with vision capabilities, but you should expect a small quality degradation.

Notes: I used unsloth's uploads of these models for convenience, since they include some extra files and configs too. I didn't name this "-Vision" because it was not trained or finetuned after weight rebase, and in case a future version by mistralai has vision.

The code will be released soon.

Evaluation

Evaluation was performed on 7 benchmarks using lm_eval and sglang. Scripts and other details will also be released with the code. This is not a comprehensive evaluation, and it's not directly comparable to the official benchmark numbers from Mistral, the goal was to approximate quality degradation. Make sure to test on your own downstream tasks!

TasksMetricDevstral-Small-2507Devstral-Small-2507-rebasedRelative Loss (%)Relative Stderr (%)
arcchallengechatexact_match0.92920.92830.10%±0.81%
eq_bencheqbench72.337673.7481-1.95%±3.52%
gsm8kexact_match0.86430.8620.27%±1.09%
gsm8kexact_match0.86050.85670.44%±1.10%
ifevalinstlevelloose_acc0.66310.65950.54%N/A
ifevalinstlevelstrict_acc0.60670.60190.79%N/A
ifevalpromptlevelloose_acc0.56190.55451.32%±3.81%
ifevalpromptlevelstrict_acc0.49170.48611.14%±4.37%
mbpppassat10.1180.1125.08%±12.20%
mmlu_proexact_match0.57860.579-0.07%±0.76%
triviaqaexact_match0.70750.70680.10%±0.48%