GreenBitAI/Qwen3.6-35B-A3B-8bit-paged

Qwen3.6-35B-A3B-8bit-paged

Qwen3.6-35B-A3B, quantized to 8 bits and packaged so that a machine which cannot hold the whole model in memory can still run it.

The model is Qwen/Qwen3.6-35B-A3B. This repository changes only how the weights are laid out on disk, not what they are.

Output is bit-identical to the same weights in their ordinary layout — the same bits, not merely close. The comparison is made while the repository is built, when both layouts still exist, and its result is recorded inside the repository.

How it works

A mixture-of-experts model only uses a few experts for any given token, but a normal checkpoint keeps all of them in memory all the time. Here the expert weights are stored separately, in a layout that allows any single expert to be read on its own.

At load time the engine looks at how much memory the machine actually has:

• —Enough for the whole model — the experts are read into memory and the model runs exactly as the original does. Nothing is streamed and nothing is different.
• —Not enough — the experts stay on disk, and each layer keeps a bounded cache of the ones it is using. The rest are read as the routing asks for them.

There is nothing to configure and no separate variant to choose. One repository serves both cases.

The repository is no larger for it: the expert tensors were moved out of the safetensors files, not duplicated.

What it costs

Streaming experts from disk is slower than having them in memory. If your machine has room for the model you pay nothing at all; if it does not, this is the trade that makes the model runnable rather than impossible.

Processing a long prompt for the first time is where the difference is most visible. Continued conversations are much less affected, because the prompt cache reuses work already done.

Requirements

gbx-lm with expert paging support. Earlier versions do not recognise this format — the expert weights are not where they expect to find them.

Requirements

Memory is not a fixed figure for a paged build, and that is the point of one: it fills what fits and streams the rest from disk. On a 512 GB Mac Studio with room to spare this model settles at about 35 GB resident. A smaller machine holds less and reads more from disk -- slower, but it runs.

How much slower depends on how far the machine is from holding the experts, and on how fast its disk is. Each token routes to a few experts; the ones already in memory cost nothing to reach, and the ones that are not have to be read before that token can finish. A machine holding most of them waits rarely, one holding few waits often. We have not measured this across machine sizes and will not guess a figure: what we can say is that the model answers either way, and that the wait is the SSD's, not the model's.

Install

curl -fL -o gbx_lm-darwin-arm64.tar.gz 'https://github.com/GreenBitAI/gbx-lm/releases/latest/download/gbx_lm-darwin-arm64.tar.gz' \
  && tar -xzf gbx_lm-darwin-arm64.tar.gz gbx_lm \
  && mkdir -p "$HOME/.local/bin" \
  && mv gbx_lm "$HOME/.local/bin/gbx_lm" \
  && chmod +x "$HOME/.local/bin/gbx_lm"

gbx_lm -h

command not found means $HOME/.local/bin is not on your PATH: add it, or call the binary by its full path. The build is signed with a Developer ID and notarised, so macOS runs it without the usual detour for a downloaded binary.

Run

gbx_lm --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged

That serves an OpenAI-compatible API on port 11688, which is its default. The weights download on first use into ~/.libra/cache/models; set HF_HOME to put them elsewhere, and HF_TOKEN if you meet the Hub's rate limits for anonymous downloads.

curl http://127.0.0.1:11688/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"GreenBitAI/Qwen3.6-35B-A3B-8bit-paged","messages":[{"role":"user","content":"Hello"}]}'

gbx_lm --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged

GBX_PAGING=off disables streaming entirely, for a machine that has the memory and wants to be certain it is not being used.

Coding agents

The server speaks three wire protocols on the same port, so the tools that expect a hosted API can be pointed at this one:

Codex -- a provider in ~/.codex/config.toml:

[model_providers.gbx]
name = "gbx-lm"
base_url = "http://127.0.0.1:11688/v1"
wire_api = "responses"

and a profile in ~/.codex/gbx.config.toml:

model_provider = "gbx"
model = "GreenBitAI/Qwen3.6-35B-A3B-8bit-paged"
model_context_window = 262144

Claude Code -- ~/.claude/gbx.settings.json:

{
  "env": {
    "ANTHROPIC_BASE_URL": "http://127.0.0.1:11688",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "GreenBitAI/Qwen3.6-35B-A3B-8bit-paged",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "GreenBitAI/Qwen3.6-35B-A3B-8bit-paged"
  }
}

Both clients ask for a small model for their own background work, so every name in the settings has to be one this server is serving.

Limits

One request at a time. The expert cache is per-model state, so concurrent requests would interfere with each other. The engine detects this and serves without streaming instead — which means a machine that depends on streaming cannot serve concurrent requests.

Files

config.json, tokenizer*     as published upstream, with two additions
model.safetensors           every weight that is not an expert
experts.bin                 the expert weights
experts_index.json          their layout, and the build-time verification record

The two additions to config.json tell the engine which weight format these are and how the expert projections are quantized — the latter would otherwise be inferred from tensors that now live in experts.bin.

The draft head

The mtp/ folder carries the model's own multi-token prediction head, so speculative decoding works from this repository alone. It is off unless asked for:

GBX_QWEN35_MTP=on gbx_lm --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged
GBX_QWEN35_MTP=on gbx_lm --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged

Measured on a 512 GB Mac Studio (M3 Ultra), 96 tokens, greedy, decode timed from the first token:

Every token the head proposes is checked by the model itself, so the reply is the model's own either way; the head only saves passes over the weights.