CoolFace
Modelpublic

onnx-community/gemma-2-9b-it-ONNX-DirectML-GenAI-INT4

sourceHugging Faceupdated 1y agoView on Hugging Face
5likes
Model Card

Model Card for Model ID

Model Details

google/gemma-2-9b quantized to ONNX GenAI INT4 with Microsoft DirectML optimization.<br> Output is reformatted that each sentence starts at new line to improve readability. <pre> ... vNewDecoded = tokenizerstream.decode(newtoken) if re.fullmatch("^[\x2E\x3A\x3B]$", vPreviousDecoded) and vNewDecoded.startswith(" ") and (not vNewDecoded.startswith(" *")) : print("\n" + vNewDecoded.replace(" ", "", 1), end='', flush=True) else : print(vNewDecoded, end='', flush=True) vPreviousDecoded = vNewDecoded ... </pre>

<img src="https://zci.sourceforge.io/epub/gemmaonnx.png">

Model Description

google/gemma-2-9b quantized to ONNX GenAI INT4 with Microsoft DirectML optimization<br> https://onnxruntime.ai/docs/genai/howto/install.html#directml

Created using ONNX Runtime GenAI's builder.py<br> https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/src/python/py/models/builder.py

Build options:<br> INT4 accuracy level: FP32 (float32)

  • —Developed by: Mochamad Aris Zamroni

Model Sources [optional]

https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct

Direct Use

This is Microsoft Windows DirectML optimized model.<br> It might not be working in ONNX execution provider other than DmlExecutionProvider.<br> The needed python scripts are included in this repository

Prerequisites:<br>

  1. 1.Install Python 3.11 from Windows Store:<br> https://apps.microsoft.com/search/publisher?name=Python+Software+Foundation
  1. 1.Open command line cmd.exe
  1. 1.Create python virtual environment, activate the environment then install onnxruntime-genai-directml<br> mkdir c:\temp<br> cd c:\temp<br> python -m venv dmlgenai<br> dmlgenai\Scripts\activate.bat<br> pip install onnxruntime-genai-directml
  1. 1.Use the onnxgenairun.py to get chat interface.<br> It is modified version of "https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py".<br> The modification makes the text output changes to new line after "., :, and ;" to make the output easier to be read.

rem Change directory to where model and script files is stored<br> cd thisonnxmodel_directory<br> python onnxgenairun.py --help<br> python onnxgenairun.py -m . -v -g

  1. 1.(Optional but recommended) Device specific optimization.<br> a. Open "dml-device-specific-optim.py" with text editor and change the file path accordingly.<br> b. Run the python script: python dml-device-specific-optim.py<br> c. Rename the original model.onnx to other file name and put and rename the optimized onnx file from step 5.b to model.onnx file.<br> d. Rerun step 4.
Speeds, Sizes, Times [optional]

6 token/s in Radeon 780M with 8GB pre-allocated RAM.

Hardware

AMD Ryzen Zen4 7840U with integrated Radeon 780M GPU<br> RAM 32GB<br>

Software

Microsoft DirectML on Windows 10

Model Card Authors [optional]

Mochamad Aris Zamroni

Model Card Contact

https://www.linkedin.com/in/zamroni/