Modelpublic

yomie4343/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-mixed-3-8bit

sourceHugging Faceotherupdated 9d agoView on Hugging Face

0likes787downloads

Model Card

Qwen3.8-Flash-Next-Uncensored — MLX-Serve mixed 3/8-bit

An experimental MLX-Serve mixed-precision pack derived from orcarouter/Qwen3.8-Flash-Next-Uncensored. Routed MoE expert weights use 3-bit affine quantization (group size 64); the remaining eligible projections use 8-bit quantization. The token embeddings and the separate ngram_table.bin remain 4-bit. The MTP head and vision files are included.

This pack is for the patched mlx-serve runtime used in the validation. It is not a generic Transformers, mlx-lm, or mlx-vlm checkpoint.

The matching V2 runtime patch and validation manifests are in `engine/README.md`.

Which variant should I use?

Two MLX-Serve-specific mixed-precision variants are available:

Variant	Positioning	Routed MoE experts	Other eligible projections	Token embeddings	N-gram table	Approx. pack size
4/8-bit	Recommended default / quality-oriented	4-bit	8-bit	4-bit	4-bit	~107 GB
3/8-bit	Lower-memory experimental variant	3-bit	8-bit	4-bit	4-bit	~86 GB

Choose the 4/8-bit variant as the default starting point when memory capacity is sufficient. It uses less aggressive quantization for the routed MoE expert weights.

Choose the 3/8-bit variant when reducing model memory or storage requirements is more important. Only the routed MoE expert weights are reduced to 3-bit; the token embeddings and N-gram table remain 4-bit, and the other eligible projections remain 8-bit.

These labels describe the intended trade-off between the two packs. They are not a claim that the 4/8-bit variant is universally more accurate, or that the 3/8-bit variant has universally equivalent quality or speed. Measured quality and performance are workload-, context-, runtime-, and hardware-dependent.

Runtime compatibility These are specialized checkpoints for the patched mlx-serve runtime used by this repository. They are not generic Transformers, `mlx-lm`, or `mlx-vlm` checkpoints. Do not rely on Hugging Face's automatically generated mlx-lm / mlx-vlm usage examples for these packs; follow this repository's engine/README.md and runtime instructions instead.

Validation notes

Results reported for each variant should be interpreted only under the conditions documented on that variant's model card. Scores obtained with different task sets, runtime revisions, generation limits, or evaluation procedures should not be compared directly.

The 3/8-bit variant has additionally been validated on a Mac Studio M3 Ultra with 96 GB unified memory using its matching patched MLX-Serve V2 runtime. In that workload, it reduced measured MLX peak memory relative to the tested 4/8-bit reference. This is a workload-specific observation, not a general memory or quality guarantee.

Download

hf download yomie4343/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-mixed-3-8bit \
  --local-dir ./qwen-flash-next-mixed-3-8bit

Requirements

Use an Apple Silicon Mac with the matching patched mlx-serve V2 runtime and keep the complete pack on a fast local SSD. The model files alone are not a drop-in installation of the runtime patch.

Runtime

Use the V2 engine with the 3-bit kernel and these environment variables:

export MLX_SERVE_Q3_PACK=8
export MLX_SERVE_Q3_BYTES=1
export MLX_SERVE_NGRAM_WARM=0
export MLX_SERVE_CACHE_LIMIT=2147483648
./zig-out/bin/mlx-serve \
  --model /absolute/path/to/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-mixed-3-8bit \
  --serve --host 127.0.0.1 --port 11234 --ctx-size 204800 \
  --kv-quant 8 --prefill-chunk 1024 --no-pld

The model pack is about 86 GB on disk, including the approximately 32 GB 4-bit n-gram table. Keep the complete directory on a fast local SSD.

Validation on Mac Studio M3 Ultra, 96 GB, 2026-09-18 JST

The patched engine reached practical speed parity with the existing 4/8-bit pack: across approximately 3K, 12K, and 47K input tokens, median prefill and decode differences stayed below 0.4% in the repeated comparison. This is a workload-bounded comparison, not a claim of universal statistical equivalence.

Actual 127,744–200,094-token inputs passed the fixed JSON and tool-retrieval checks in all 6/6 runs for this pack. Its cumulative MLX peak in that phase was 63.774 GB, about 14.7 GB below the 4/8-bit reference.

The original 25-task Hermes/TypeScript comparison was independently graded at 72/75 (three repeated runs of the same 25 tasks). The three failures were the same binary-search-tree implementation type error (missing return values). A separate fixed-8192-output supplement for the two tasks affected by Hermes' adaptive output cap passed 6/6 across this pack; that supplement is reported separately and is not pooled with the 72/75 score.

Limitations

This is an experimental quantization and runtime combination. The validation sets are fixed and do not guarantee general coding quality, reasoning quality, or performance on other hardware. The source model is uncensored and can produce unsafe, incorrect, or offensive output. Use least-privilege tools and application-level safeguards.

The Qwen Community License 1.0 is retained; see LICENSE and review its terms before redistribution or commercial use.