0xKitkat/Agnes-3.0-Flash-GGUF

Agnes-3.0-Flash GGUF

Follow [@procrastiness on Twitter/X](https://twitter.com/procrastiness) for more model releases and updates.

GGUF quantizations of the original Agnes-AI/Agnes-3.0-Flash, released by 0xKitkat. This is the normal model, with no abliteration or fine-tuning. Approximately 33.1B parameters; all 72 decoder blocks and the original vision tower are retained.

The original Apache-2.0 license is preserved. Downloads and local use require no hosted API subscription; you supply the hardware and electricity. The source revision is 8f0c484c363cdda8384195be4a5f7730f3915bde.

Available validated quants: Q4KM, Q5KM, Q6K, Q80. All planned quants are published.

<!-- GGUF-USAGE-BEGIN -->

Recommended starting setup

Start with Q4_K_M, a 4,096-token context, one concurrent request, and thinking disabled. This is the configuration closest to the functional checks reported below. Use the embedded chat template; do not select a generic ChatML or Llama template manually.

For normal chat, use the upstream sampling defaults: temperature 1.0, top-p 0.95, top-k 20. For repeatable troubleshooting or checking exact answers, use temperature 0. Start with 512 output tokens for short answers or 2,048 for longer responses, keeping prompt + image tokens + output within the configured context.

Choose a quant and hardware

Download one text GGUF. Add the F16 vision projector only if you want image input. The projector is shared by all quants in this release.

The validation table lists exact sizes for published files. A quant being listed here does not mean its upload has finished; check the available-files list and the Files tab.

These are starting recommendations, not exact minimums. Weight-file size excludes runtime buffers, recurrent state, attention cache, images, and other applications. A 16 GB machine is a poor fit for this approximately 33B model. CPU-only operation benefits from 48–64 GB RAM when other applications are running.

1. Install llama.cpp and the download tool

Use a recent llama.cpp build with Qwen3.5 text and vision support. The release was tested with commit 56381e407c0ccfb3a6f71e668a27a901001d22ce. Older bundled runtimes can reject the model architecture or load it incorrectly.

Windows

Install Python 3.10+ if needed, then open PowerShell:

winget install llama.cpp
python -m pip install --upgrade huggingface_hub requests
llama-server --version

Open a new terminal after installing if commands are not found. For NVIDIA acceleration, use a compatible CUDA build from the official llama.cpp releases, extract the complete archive, and keep its DLLs alongside llama-server.exe. From that folder, replace llama-server in the commands below with .\llama-server.exe. Confirm the startup log detects your GPU; installing a package alone does not establish which backend it uses.

macOS

With Homebrew and Python installed:

brew install llama.cpp
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade huggingface_hub requests
llama-server --version

Ubuntu / Debian: reproducible NVIDIA build

Install a compatible NVIDIA driver and CUDA Toolkit first; nvidia-smi and nvcc --version should work. Then:

sudo apt-get update
sudo apt-get install -y git cmake build-essential libcurl4-openssl-dev python3-venv
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 56381e407c0ccfb3a6f71e668a27a901001d22ce
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4 --target llama-server
export PATH="$PWD/build/bin:$PATH"
cd ..
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade huggingface_hub requests

For a CPU-only build, configure with -DGGML_CUDA=OFF and start the server with -ngl 0. Other backends and installation options are covered in the llama.cpp installation guide and build guide.

2. Download the model

Run these in the directory where you want the agnes folder. The public download does not require a paid account or hosted inference subscription.

hf download 0xKitkat/Agnes-3.0-Flash-GGUF Agnes-3.0-Flash-Q4_K_M.gguf --local-dir agnes

For image input, also download:

hf download 0xKitkat/Agnes-3.0-Flash-GGUF mmproj-Agnes-3.0-Flash-F16.gguf --local-dir agnes

For another published quant, replace Q4_K_M in the filename with Q5_K_M, Q6_K, or Q8_0. Rerun the same download command after an interruption. Allow disk space for the chosen file plus the approximately 0.93 GB projector if used.

SHA-256 checksums are recorded in release-manifest.json. To calculate a local checksum:

Get-FileHash agnes/Agnes-3.0-Flash-Q4_K_M.gguf -Algorithm SHA256

On Linux use sha256sum agnes/Agnes-3.0-Flash-Q4_K_M.gguf; on macOS use shasum -a 256 with the same path.

3. Start the local server

Text-only: single GPU or automatic fitting

llama-server -m agnes/Agnes-3.0-Flash-Q4_K_M.gguf --alias agnes -c 4096 --parallel 1 --fit on --jinja --flash-attn on --batch-size 256 --ubatch-size 128 --threads 6 --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080

Open http://127.0.0.1:8080 for the built-in chat UI. Keep the terminal running. The API is at http://127.0.0.1:8080/v1; its model name is `agnes`. This local server does not need an API key. If a client requires a nonempty key field, use local.

The command lets llama.cpp choose GPU offloading with --fit on. If it runs out of GPU memory, specify a smaller layer count, for example -ngl 20, and adjust from there. Use -ngl 0 for CPU-only inference. Six CPU threads is a starting value used in our checks; tune it for your CPU.

Text and images: tested dual-GPU layout

llama-server -m agnes/Agnes-3.0-Flash-Q4_K_M.gguf --mmproj agnes/mmproj-Agnes-3.0-Flash-F16.gguf --alias agnes -c 4096 --parallel 1 -ngl 99 --split-mode layer --tensor-split 1,1 --jinja --flash-attn on --batch-size 256 --ubatch-size 128 --threads 6 --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080

-ngl 99 requests all eligible layers on the GPUs; it is not a claim that the model has 99 layers. --tensor-split 1,1 distributes layers between two GPUs. On one GPU, use the first command and add --mmproj agnes/mmproj-Agnes-3.0-Flash-F16.gguf. For larger quants, lower the GPU layer count or let automatic fitting choose it.

Thinking and context settings

The API examples below explicitly disable thinking, matching release testing. To make this the server default for the web UI too, set LLAMA_ARG_CHAT_TEMPLATE_KWARGS before starting the server:

# PowerShell
$env:LLAMA_ARG_CHAT_TEMPLATE_KWARGS = '{"enable_thinking":false}'

# Bash / zsh
export LLAMA_ARG_CHAT_TEMPLATE_KWARGS='{"enable_thinking":false}'

For experimental thinking-on use, send "chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "xhigh"} in the request. The source template accepts low, medium, and xhigh effort (the actual pinned template uses xhigh, even though the upstream card describes high). Allocate more output tokens and context for reasoning. This release's measured results do not validate reasoning-on quality, tool calling, video, or the advertised 262,144-token context. Start at 4K; try 8K only after confirming adequate memory and prompt-plus-output space.

4. Use the API from Python

Save as chat_agnes.py, then run python chat_agnes.py while the server is running:

import unicodedata
import requests

prompt = unicodedata.normalize("NFC", "Explain how a rainbow forms in three sentences.")
response = requests.post(
    "http://127.0.0.1:8080/v1/chat/completions",
    json={
        "model": "agnes",
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20,
        "max_tokens": 512,
        "chat_template_kwargs": {"enable_thinking": False},
        "stream": False,
    },
    timeout=1800,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"]["content"])

For deterministic checks, change temperature to 0 and add "seed": 20260912. Sampling defaults above come from the original model; they are a starting point, not a new tuning benchmark.

Ask about a local image

Start the server with the projector. Save this as image_agnes.py, place an image.jpg alongside it, and run python image_agnes.py:

import base64
from pathlib import Path
import requests

encoded = base64.b64encode(Path("image.jpg").read_bytes()).decode("ascii")
response = requests.post(
    "http://127.0.0.1:8080/v1/chat/completions",
    json={
        "model": "agnes",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the main objects in this image."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encoded}"}},
            ],
        }],
        "temperature": 0,
        "max_tokens": 256,
        "chat_template_kwargs": {"enable_thinking": False},
        "stream": False,
    },
    timeout=1800,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"]["content"])

For PNG input, change the file path and use data:image/png;base64,. Begin with one modest-size image; image tokens also consume context. The API format follows llama.cpp's multimodal server documentation.

Troubleshooting

Other applications must bundle a sufficiently recent compatible backend. LM Studio and Ollama were not part of this release's validation; the commands above use llama.cpp directly. <!-- GGUF-USAGE-END -->

Validation

These are small functional tests, not comprehensive capability benchmarks. Native-reference checks compare next-token distributions on 16 prompts with a layer-streamed upstream calculation using FP32 computation and FP16 residual storage. All generation checks use thinking disabled. Per-quant reports and SHA-256 hashes are included. Long-context limits, video, reasoning-on quality, and tool calling are not exhaustively tested here.

WikiText perplexity uses eight 512-token chunks from a pinned WikiText-2 test corpus. Each normal quant must stay within 5% of the unmodified Q4 baseline. This is a subset loss check, not a full-corpus benchmark.

Q4KM, Q5KM and Q6K use a 32-chunk importance matrix computed from the unmodified Q80 model. Q8_0 is quantized directly from the original BF16 weights. The projector is F16 and is shared by all quants.

Architecture mapping

Agnes uses the same attention computation as the supported Qwen3.5 GGUF graph. Its additional parallel SwiGLU branch is preserved exactly by concatenating both branches' gate/up matrices and concatenating their down matrices on the input dimension. This is an algebraic graph conversion, not a trained modification. MTP speculative decoding weights are omitted from GGUF. The vision tower maps to the Qwen3.5 vision projector graph. Structural, rotary, tokenizer, and FP32 next-token equivalence tests are included.

The upstream tokenizer applies NFC Unicode normalization; stock llama.cpp's Qwen3.5 tokenizer does not. For exact agreement on decomposed Unicode, normalize text with unicodedata.normalize("NFC", text) before rendering the chat prompt. Already-normalized English and Chinese text is unaffected.

Original model and implementation: Agnes AI. Build code was generated with AI assistance and tested as reported. See the original license and upstream model card for attribution.