Modelpublic

philschmid/quantized-distilbert-banking77

sourceHugging Faceupdated 4y agoView on Hugging Face

0likes55downloads

Model Card

Quantized-distilbert-banking77

This model is a statically quantized version of optimum/distilbert-base-uncased-finetuned-banking77 on the banking77 dataset.

The model was created using the optimum-static-quantization notebook.

It achieves the following results on the evaluation set:

Accuracy

—Vanilla model: 92.5%
—Quantized model: 92.24%

The quantized model achieves 99.72% accuracy of the fp32 model

Latency

Payload sequence length: 128 Instance type: AWS c6i.xlarge

latency	vanilla transformers	quantized optimum model	improvement
p95	75.69ms	26.75ms	2.83x
avg	57.52ms	24.86ms	2.31x

How to use

python

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import pipeline, AutoTokenizer

model = ORTModelForSequenceClassification.from_pretrained("philschmid/quantized-distilbert-banking77")
tokenizer = AutoTokenizer.from_pretrained("philschmid/quantized-distilbert-banking77")

remote_clx = pipeline("text-classification",model=model, tokenizer=tokenizer)

remote_clx("What is the exchange rate like on this app?")