Modelpublic

Ringoacid/Qwen2.5-0.5B-onnx-attention-int8

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes8downloads

Model Card

Qwen2.5-0.5B ONNX (int8) — Attention Visualization Edition

This is an ONNX-converted, 8-bit dynamically quantized version of `Qwen/Qwen2.5-0.5B`, packaged for in-browser inference via ONNX Runtime Web with the WebGPU execution provider.

Original model: `Qwen/Qwen2.5-0.5B` (Apache-2.0)

Files

File	Purpose	Size
`analyze_int8.onnx` (+ `.data`)	Single forward pass with per-layer attention weights as graph outputs (eager attention, no KV cache) — used by the visualizer's "analyze" mode	~476 MB
`generate_int8.onnx` (+ `.data`)	Merged prefill+decode model with KV cache (SDPA, no attention output) — used by the visualizer's "generate" mode	~474 MB
`tokenizer.json`, `vocab.json`, `merges.txt`, etc.	Qwen2 BPE tokenizer (Transformers.js compatible)	~16 MB

I/O schema

`analyze_int8.onnx`

—Inputs: input_ids[B,S], attention_mask[B,S] (both int64)
—Outputs: logits[B,S,151936], attentions.0..23[B,14,S,S]

`generate_int8.onnx`

—Inputs: input_ids[B,S], attention_mask[B,P+S], position_ids[B,S], and 24x past_key_values.{i}.{key,value}[B,2,P,64]
—Outputs: logits[B,S,151936], 24x present.{i}.{key,value}[B,2,P+S,64]

Where B=batch (1), S=current sequence length, P=past sequence length.

Quantization

—Type: dynamic, weight-only QInt8, per-channel
—Tool: onnxruntime.quantization.quantize_dynamic
—The top-1 next token matches between fp32 and int8 for short prompts; some degradation on top-5 ranks is expected.

Usage (browser, WebGPU)

import * as ort from 'onnxruntime-web';
const base = 'https://huggingface.co/Ringoacid/Qwen2.5-0.5B-onnx-attention-int8/resolve/main';
const sess = await ort.InferenceSession.create(`${base}/analyze_int8.onnx`, {
  executionProviders: ['webgpu', 'wasm'],
  graphOptimizationLevel: 'all',
});

License

Apache-2.0, inherited from the base model.