CoolFace
Modelpublic

Mr-Corentin/myhaiku-gemma-3-270m-it-onnx

sourceHugging Facemitupdated 11mo agoView on Hugging Face
0likes23downloads
Model Card

myhaiku-gemma-3-270m-it (ONNX)

This repository contains ONNX exports of the fine-tuned Gemma model: 👉 Mr-Corentin/myhaiku-gemma-3-270m-it

These ONNX models are optimized for efficient inference with ONNX Runtime, suitable for deployment on CPU, GPU, or WebGPU backends.

Model Summary

PropertyDescription
Base modelgoogle/gemma-3-270m-it
Fine-tuned modelMr-Corentin/myhaiku-gemma-3-270m-it
Export typeONNX (fp32, fp16, q8, q4, q4f16)
FrameworkONNX Runtime
Use caseText generation (Haiku generation)

Available Model Variants

FilePrecisionNotes
onnx/model.onnxFP32Full precision reference
onnx/model_fp16.onnxFP16Recommended for GPU
onnx/model_q8.onnxINT48-bit weights
onnx/model_q4.onnxINT44-bit weights, compact but lower fidelity
onnx/model_q4f16.onnxINT4 weights + FP16 activations

All models share the same tokenizer and configuration files.

🚀 Usage Example

python
from transformers import AutoConfig, AutoTokenizer, GenerationConfig
import onnxruntime
import numpy as np

save_path = "path/to/myhaiku-gemma-3-270m-it-onnx"
config = AutoConfig.from_pretrained(save_path)
generation_config = GenerationConfig.from_pretrained(save_path)
tokenizer = AutoTokenizer.from_pretrained(save_path)

model_path = f"{save_path}/onnx/model_fp16.onnx"
session = onnxruntime.InferenceSession(model_path)

prompt = "Write a haiku about the beauty of autumn."
messages = [
    {"role": "system", "content": "You are a haiku generator. Reply with exactly three short lines, no extra text."},
    {"role": "user", "content": prompt},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="np")
input_ids = inputs["input_ids"]
attention_mask = inputs["attention_mask"]

# Run inference (example for 1 token generation)
logits = session.run(None, {"input_ids": input_ids, "attention_mask": attention_mask})[0]
next_token = np.argmax(logits[:, -1, :], axis=-1)
print(tokenizer.decode(next_token[0]))

License

This model and code are released under the MIT license.

Credits