Modelpublic

vistralis/FLUX.2-klein-4b-INT8-transformer-quants

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face

1likes120downloads

Model Card

FLUX.2-klein-4b-INT8-transformer-quants

INT8 (W8A8) quantization variants for FLUX.2-klein-4B (step-distilled 4B parameters).

This repository contains multiple INT8 quantization variants for experimentation and comparison.

Status: Static/Max variants (int8-per-row, int8-per-tensor) are available now. SmoothQuant variants are pending and will be added when ready.

Variant	Algorithm	Scale Mode	Status	Checkpoint
int8-per-row	static	per-row	✅ Available	`flux-2-klein-4b-int8-per-row.safetensors`
int8-per-tensor	static	per-tensor	✅ Available	`flux-2-klein-4b-int8-per-tensor.safetensors`
int8-smoothquant-per-row	smoothquant	per-row	🔜 Pending	`flux-2-klein-4b-int8-smoothquant-per-row.safetensors`
int8-smoothquant-per-tensor	smoothquant	per-tensor	🔜 Pending	`flux-2-klein-4b-int8-smoothquant-per-tensor.safetensors`

Quantization Details

All variants use NVIDIA TensorRT Model Optimizer (ModelOpt) INT8 (W8A8) quantization:

Property	Value
Framework	NVIDIA ModelOpt
Calibration	768 prompts (256 T2I, 256 editing, 256 composition), 4 steps each
Weight Quantization	INT8 symmetric — per-row or per-tensor depending on variant
Activation Quantization	Dynamic per-row (quantized on-the-fly at inference, one scale per token)
Preserved Layers	Embedder layers (timeembed, contextembedder, x_embedder) and output projection kept in BF16

Algorithm × Scale Mode

	Per-Row	Per-Tensor
Static (Max)	`int8-per-row` ✅	`int8-per-tensor` ✅
SmoothQuant	`int8-smoothquant-per-row` 🔜	`int8-smoothquant-per-tensor` 🔜

Algorithm:

—Static (Max): Standard INT8 quantization with calibrated min/max ranges
—SmoothQuant (pending): Migrates quantization difficulty from activations to weights for better accuracy

Scale Mode:

—Per-Row: Independent scale per output channel (finer granularity, higher accuracy)
—Per-Tensor: Single scale per tensor (faster, lower memory, slightly reduced accuracy)

Note: In all variants, input activations are always quantized dynamically per-row at inference time (one scale per token). The scale mode above refers to the weight quantization granularity.

Evaluation Results

Compared against BF16 baseline using identical prompts, seeds, and resolution.

Overall Metrics

Variant	CLIP ↑	LPIPS ↓	PSNR ↑	MSE ↓	FID ↓
BF16 (baseline)	0.6518	—	—	—	—
FP8 (reference)	0.6522	0.0356	27.60	210.14	22.94
int8-per-row	0.6526	0.0253	29.23	142.27	15.76
int8-per-tensor	0.6517	0.0447	26.25	254.55	24.83
int8-smoothquant-per-row	—	—	—	—	—
int8-smoothquant-per-tensor	—	—	—	—	—

Text-to-Image

Variant	CLIP ↑	LPIPS ↓	PSNR ↑
FP8 (reference)	0.6452	0.0480	24.43
int8-per-row	0.6458	0.0343	25.89
int8-per-tensor	0.6466	0.0621	23.23
int8-smoothquant-per-row	—	—	—
int8-smoothquant-per-tensor	—	—	—

Dramatic chiaroscuro portrait of a cellist mid-performance, single spotlight from above, instrument bow caught in motion blur, concert hall darkness

[image]

Stained glass window design depicting the four elements, lead came outlines, rich jewel tones of ruby, sapphire, emerald, and topaz

[image]

Editing

Variant	CLIP ↑	LPIPS ↓	PSNR ↑
FP8 (reference)	0.6467	0.0178	32.28
int8-per-row	0.6465	0.0150	33.69
int8-per-tensor	0.6447	0.0256	30.21
int8-smoothquant-per-row	—	—	—
int8-smoothquant-per-tensor	—	—	—

Base: A bicycle leaning against a stone wall in a village Edit: Transform the village into an underwater coral reef scene, the bicycle covered in barnacles and sea anemones, fish swimming around

[image]

Base: A food truck parked on a city street at noon Edit: Change the street to a Venice canal with the food truck floating on a gondola platform, evening golden hour lighting

[image]

Composition

Variant	CLIP ↑	LPIPS ↓	PSNR ↑
FP8 (reference)	0.6649	0.0410	26.08
int8-per-row	0.6654	0.0267	28.10
int8-per-tensor	0.6638	0.0465	25.32
int8-smoothquant-per-row	—	—	—
int8-smoothquant-per-tensor	—	—	—

Create a zen garden where the raked sand patterns flow into and around a giant ramen bowl as the central stone

[image]

A clockwork mechanical wolf made of brass gears howling at the full moon on the snowy ridge, steam rising from its joints

[image]

Usage

🚧 Code release coming soon. A pip-installable loader library is in preparation.

In the meantime, these checkpoints can be tested with ComfyUI using the ComfyUI-Flux2-INT8 custom node. Per-row quantization support is available via PR #24.

Technical Details

Property	Value
Base Model	FLUX.2-klein-4B
Parameters	4B
Quantization	INT8 (W8A8) via NVIDIA ModelOpt
Calibration	768 prompts (256 per task), 4 steps each
Activation Quantization	Dynamic per-row (quantized on-the-fly at inference)
Preserved Layers	Embedder layers and output projection kept in BF16
Inference Steps	4
Guidance Scale	1.0

License

This model inherits the license from the base model: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).