Modelpublic

groxaxo/Qwen-AgentWorld-35B-A3B-GGUF

sourceHugging Faceupdated 1mo agoView on Hugging Face

4likes489downloads

Model Card

Qwen-AgentWorld-35B-A3B GGUF

Overview

Qwen-AgentWorld-35B-A3B-GGUF is a GGUF release for llama.cpp-compatible runtimes and local inference, published by `groxaxo`. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

Field	Details
Format	GGUF
Source / base	`Qwen/Qwen-AgentWorld-35B-A3B`
Intended task	the task described by the included configuration and documentation
License	the license declared in the repository files

What is included

—*.gguf (6 files)
—Additional configuration, tokenizer, processor, or shard files (6 visible artifacts total)

Quick start

llama.cpp

Download a .gguf file that fits your available memory, then run it with a current llama.cpp build:

bash

llama-cli \
  -m /path/to/model.gguf \
  -p "Write a concise technical summary."

For vision or any-to-any models, download the matching multimodal projection file when one is provided and follow the source model's modality-specific instructions.

Compatibility and responsible use

—Use a runtime that explicitly supports this format, architecture, and modality.
—Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
—Review the source model card and license before redistribution or deployment.
—Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
—Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

GGUF quantizations of Qwen/Qwen-AgentWorld-35B-A3B generated with llama.cpp.

—Architecture: qwen35moe (35B params, 256 experts / 8 active, A3B)
—Context length: 262 144
—Source: BF16 GGUF converted via convert_hf_to_gguf.py
—Importance matrix (coding.imatrix.gguf) generated from a curated coding calibration text (1 840 samples), 128 chunks, ctx 4 096.

Files

File	Size (GiB)	BPW	Use
`Qwen-AgentWorld-35B-A3B-BF16.gguf`	64.61	16.01	reference / highest fidelity
`Qwen-AgentWorld-35B-A3B-Q8_0.gguf`	34.37	8.52	near-lossless, needs >24 GB VRAM with all layers offloaded
`Qwen-AgentWorld-35B-A3B-Q6_K.gguf`	26.56	6.58	very high quality, ~20 GB VRAM
`Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf`	19.71	4.88	balanced quality / size, ~14 GB VRAM
`Qwen-AgentWorld-35B-A3B-Q2_K.gguf`	12.05	2.99	smallest, ~9 GB VRAM, quality trade-off
`Qwen-AgentWorld-35B-A3B-coding.imatrix.gguf`	0.18	—	importance matrix for finer quant recipes (Tensor-type overrides).

Plain quantization passes (no recipe overrides) only — these avoid the std::bad_alloc triggered by --tensor-type-file + imatrix on this specific MoE architecture in the current llama.cpp build (8194 / 1179bfc82). The imatrix file is still provided for users who want to mix types via --tensor-type-file.

Recommended inference

llama.cpp/build/bin/llama-server \
    -m Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf \
    -ngl 999 --ctx-size 8192 -b 2048 -ub 512 -np 1 \
    --temp 0.6 --top-p 0.95 --top-k 20 \
    -fa --host 0.0.0.0 --port 8080