Hal0ai/FastContext-Hal0-4B-ROCmFP4
<p align="center"> <img src="hal0-banner.png" alt="hal0" width="420"/> </p>
FastContext-Hal0-4B โ ROCmFP4 (STRIX_LEAN)
A 4-bit ROCmFP4 quantization of `microsoft/FastContext-1.0-4B-SFT`, a lightweight repository-exploration subagent (Qwen3-4B backbone) for LLM coding agents.
Quantized and validated on AMD Strix Halo (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151) using `hal0ai/amd-strix-halo-toolboxes` ๐ ๏ธ.
### โ ๏ธ Read this first โ special runtime required This file uses the experimental `Q4_0_ROCMFP4` GGUF tensor format. It is NOT loadable by stock llama.cpp, Ollama, LM Studio, or any standard GGUF runtime. It runs only in the `charlie12345/rocmfp4-llama` fork. ROCmFP4 is a custom Codebook10 / finite-UE4M3 layout โ it is not MXFP4 or NVFP4.What's in this repo
STRIX_LEAN is a tensor-aware preset: norms stay f32, sensitive tensors keep higher precision, and the bulk of the weights use the dual/fast ROCmFP4 layouts.
Why ROCmFP4 here
On Strix Halo, token generation is memory-bandwidth-bound, so 4-bit weights decode much faster than BF16 while keeping quality intact for tool-calling.
Performance (llama-bench, ROCm0, FlashAttention on, Radeon 8060S)
Tool-calling quality (server-test-function-call.py, 5 multi-turn cases, greedy temp 0)
In every case both models selected and ordered the correct tools โ the only failures were "no final summary produced" after correct tool use, a stopping quirk shared by the BF16 source (not a quantization artifact). Takeaway: FP4 introduced no measurable tool-calling regression. A 5-case harness can't rank models finely, so read this as "quality preserved," not "FP4 > BF16."
How to run
Build the fork for your AMD GPU (see its README), then:
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
./build-strix-rocmfp4/bin/llama-server \
-m FastContext-4B-ROCmFP4-STRIX_LEAN.gguf \
-dev ROCm0 -ngl 999 -c 262144 -fa on --jinjaFor scripted/non-interactive generation use llama-completion (this fork's llama-cli is interactive-only and rejects -no-cnv). FastContext supports up to 262K context.
How it was made
# 1. HF safetensors -> BF16 GGUF
python convert_hf_to_gguf.py ./FastContext-1.0-4B-SFT --outtype bf16 --outfile fc-bf16.gguf
# 2. BF16 -> ROCmFP4 (same fork binary the server uses)
llama-quantize fc-bf16.gguf FastContext-4B-ROCmFP4-STRIX_LEAN.gguf Q4_0_ROCMFP4_STRIX_LEANLicense & attribution
- Weights derive from `microsoft/FastContext-1.0-4B-SFT` โ MIT.
- Backbone: `Qwen/Qwen3-4B-Instruct-2507` โ Apache-2.0.
- Quantization format & tooling: `charlie12345/rocmfp4-llama`.
This repository redistributes a quantized derivative under the terms of the upstream MIT license.
About hal0ai
Built and benchmarked with [hal0ai](https://github.com/hal0ai) โ local-first AI agent infrastructure tuned for AMD Strix Halo. The `amd-strix-halo-toolboxes` ship ready-to-run ROCm + ROCmFP4 container images so you can quantize and serve large models on a single unified-memory APU. If you're running agents on AMD silicon, come say hi. ๐
A note from the author ๐
This is my first time doing any kind of custom model quantization or training โ this release is very much a learning project. So if you spot something I got wrong, or have tips on presets, calibration, or quality testing, I'd genuinely appreciate the feedback โ open a Community discussion and let me know.
I made this to run as a slot in [hal0](https://github.com/hal0ai), alongside the main agent โ a small, fast repository-exploration subagent that ROCmFP4 lets me keep resident on the Strix Halo without crowding out the bigger models sharing the same unified memory.
If you're tinkering with local agents on AMD hardware, come check out hal0 โ would love to see what you build. ๐