xpuenabler/gpt-oss-15.5b-23E-SFT-v6-gpu

gpt-oss-15.5b-23E-SFT-v6-gpu

GPU-only stateful OpenVINO IR re-exported from xpuenabler/gpt-oss-15.5b-23E-SFT, designed to be loaded with openvino_genai.LLMPipeline(model_dir, "GPU") and to reduce GPU TTFT.

This is the v5-gpu recipe plus the MoE expert ReduceSum decomposition baked into the export (previously applied as a separate post-processing "reducesum-repaired" step). All other GPU stability rewrites are preserved.

Recipe

Quantization is a single NNCF pass with ignored_scope=[".*mlp\\.experts.*", ".*mlp\\.router.*"]. MoE experts are kept FP16 because Intel GPU lacks a compressed-weight kernel for the 3D batched MoE matmul; quantizing them would force a large dequant scratch buffer at compile time (the v4-style INT8 MoE caused a 727 MiB USM Host allocation failure on Lunar Lake).

Graph rewrites

1. 1.SDPA decomposition — 6-input ScaledDotProductAttention (with attention sink) is expanded into matmul → scale → +mask → concat(sink) → softmax primitives → slice → matmul. The GPU program_builder rejects the 6-input SDPA with Invalid inputs count (6).
2. 2.Softmax decomposition — every Softmax is expanded into ReduceMax → Subtract → Exp → ReduceSum → Divide. This removes the anchor that OpenVINO MOC SDPAFusion uses to re-fuse the decomposed attention chain back into a 6-input SDPA at GPU compile time.
3. 3.Expert `ReduceSum` decomposition (new in v6) — every mlp.experts/.../aten::sum/ReduceSum(axis=0) over the 23-expert dimension is rewritten as Split(axis=0, num_splits=23) → 23×Squeeze → Add chain. Without this, the GPU MoE extension rejects the 4D reduce.
4. 4.Defensive FP32 → FP16 — any FP32 constant >1 MB is rewritten as Constant(fp16) → Convert(fp32). Catches anything dtype=fp16 missed.

Verified after build

SDPA=0  Softmax=0  expert-ReduceSum=0
stateful: ReadValue=48 Assign=48 beam_idx=True
openvino_model.bin = 29.00 GB

CPU smoke test (openvino_genai.LLMPipeline(..., "CPU"), max_new_tokens=24) produced coherent text on prompt "Hello, who are you?".

Usage

import openvino_genai as ov_genai
pipe = ov_genai.LLMPipeline("xpuenabler/gpt-oss-15.5b-23E-SFT-v6-gpu", "GPU")
print(pipe.generate("Hello, who are you?", max_new_tokens=64))

Reproducibility

build_v6_gpu.py in this repo is the exact script that produced the IR.