noneUsername/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token

Note: This model is no longer the optimal W8A8 quantization, please consider using a better quantization model I made later: noneUsername/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token-better

My first quantization uses the quantization method provided by vllm:

https://docs.vllm.ai/en/latest/quantization/int8.html

NUMCALIBRATIONSAMPLES = 2048

MAXSEQUENCELENGTH = 8192

smoothing_strength=0.8

I will verify the validity of the model and update the readme as soon as possible.

edit: The performance in my ERP test was comparable to Mistral-Nemo-Instruct-2407-GPTQ-INT8, which I consider a successful quantization.

vllm (pretrained=/root/autodl-tmp/Mistral-Nemo-Instruct-2407,addbostoken=true,tensorparallelsize=2,maxmodellen=4096,gpumemoryutilization=0.85,swapspace=0), genkwargs: (None), limit: 250.0, numfewshot: 5, batchsize: auto |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr| |-----|------:|----------------|-----:|-----------|---|----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exactmatch|↑ |0.800|± |0.0253| | | |strict-match | 5|exactmatch|↑ |0.784|± |0.0261|

lmeval --model vllm --modelargs pretrained="/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token",addbostoken=true,dtype=half,tensorparallelsize=2,maxmodellen=4096,gpumemoryutilization=0.85,swapspace=0 --tasks gsm8k --numfewshot 5 --limit 250 --batchsize 1 vllm (pretrained=/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token,addbostoken=true,dtype=half,tensorparallelsize=2,maxmodellen=4096,gpumemoryutilization=0.85,swapspace=0), genkwargs: (None), limit: 250.0, numfewshot: 5, batchsize: 1 |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr| |-----|------:|----------------|-----:|-----------|---|----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exactmatch|↑ |0.784|± |0.0261| | | |strict-match | 5|exact_match|↑ |0.768|± |0.0268| In gsm8k, still a bit worse than the original...

lmeval --model vllm \ largs > --modelargs pretrained="/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token",addbostoken=true,dtype=half,tensorparallelsize=2,maxmodellen=4096,gpumemoryutilization=0.85,swapspace=0 \ ks hellaswag \

--tasks hellaswag \ --limit 150 \ --num_fewshot 10 \

--batch_size 1

vllm (pretrained=/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token,addbostoken=true,dtype=half,tensorparallelsize=2,maxmodellen=4096,gpumemoryutilization=0.85,swapspace=0), genkwargs: (None), limit: 150.0, numfewshot: 10, batchsize: 1 | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |---------|------:|------|-----:|--------|---|-----:|---|-----:| |hellaswag| 1|none | 10|acc |↑ |0.5800|± |0.0404| | | |none | 10|acc_norm|↑ |0.7533|± |0.0353|