CoolFace
Modelpublic

ENOT-AutoDL/gpt2-tensorrt

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face
4likes
Model Card

GPT2

This repository contains GPT2 onnx models compatible with TensorRT:

  • —gpt2-xl.onnx - GPT2-XL onnx for fp32 or fp16 engines
  • —gpt2-xl-i8.onnx - GPT2-XL onnx for int8+fp32 engines

Quantization of models was performed by the ENOT-AutoDL framework. Code for building of TensorRT engines and examples published on github.

Metrics:

GPT2-XL

TensorRT INT8+FP32torch FP16
Lambada Acc72.11%71.43%

Test environment

  • —GPU RTX 4090
  • —CPU 11th Gen Intel(R) Core(TM) i7-11700K
  • —TensorRT 8.5.3.1
  • —pytorch 1.13.1+cu116

Latency:

GPT2-XL

Input sequance lengthNumber of generated tokensTensorRT INT8+FP32 mstorch FP16 msAcceleration
646446211902.58
6412892023602.54
64256189047102.54

Test environment

  • —GPU RTX 4090
  • —CPU 11th Gen Intel(R) Core(TM) i7-11700K
  • —TensorRT 8.5.3.1
  • —pytorch 1.13.1+cu116

How to use

Example of inference and accuracy test published on github:

shell
git clone https://github.com/ENOT-AutoDL/ENOT-transformers