karmx/Nex-N2.5-mini-Mixed-Q2Q3-128K-GGUF

Nex-N2.5-mini — Mixed Q2/Q3 GGUF, vision, tested 128K configuration

GitHub companion repository: KarmSakha/Nex-N2.5-mini-Mixed-Q2Q3-128K-GGUF — reproducibility files, benchmark evidence, deployment guidance, citation information and the versioned release history.

An unofficial, calibration-assisted mixed-precision quantization of Nex-AGI's Nex-N2.5-mini, packaged by karmx for experimentation on a single RTX 5060 Ti 16 GB. Credit for the original model and training belongs to Nex-AGI and its upstream contributors. This is not an official Nex-AGI release.

What is demonstrated: the unchanged GGUF and vision projector load with 131,072 context on that card; unmodified official llama.cpp b10896 passes basic text, structured tool calling and single-image OCR. The broader test suite used a different, documented patched build.

What is not claimed: lossless compression, FP8/BF16 equivalence, no coding regression, identical speed on stock llama.cpp, guaranteed zero CPU/shared-memory paging, or universally reliable 128K/multi-image reasoning. See the failures below.

Download both files for vision

The filename retains trial so it matches the exact tested artifact. Downloadable file size is not total VRAM usage; cache, vision and compute buffers require additional memory. SHA-256 checksums are in SHA256SUMS.

hf download karmx/Nex-N2.5-mini-Mixed-Q2Q3-128K-GGUF Nex-N2.5-mini-mixed-Q2Q3-vision-trial.gguf mmproj-Nex-N2.5-mini-f16.gguf --local-dir ./nex-mini-gguf

Run with stock llama.cpp

Verified version: official b10896, commit fa6769818708afd9807b22183ccda112fd563427, Windows x86_64 CUDA 13.3. Extract the server archive and its matching CUDA runtime archive together. Their official archive hashes are recorded in STOCK-LLAMA-COMPATIBILITY.md. Other operating systems, GPUs and runtime versions were not covered by that check.

From the model directory, with llama-server on PATH:

llama-server -m Nex-N2.5-mini-mixed-Q2Q3-vision-trial.gguf --mmproj mmproj-Nex-N2.5-mini-f16.gguf --override-kv qwen35moe.block_count=int:40,qwen35moe.nextn_predict_layers=int:0 --host 127.0.0.1 --port 8083 -c 131072 -np 1 -ngl 99 -fa on -ctk q4_0 -ctv q4_0 -b 512 -ub 128 --fit off --load-mode none --lazy-mode off --alias nex-stock-test --api-key-file YOUR_API_KEY_FILE

Replace YOUR_API_KEY_FILE with a file containing your own API key. The server binds to loopback; configure your own authentication/TLS before exposing it to a network. OpenAI-compatible base URL: http://127.0.0.1:8083/v1; model alias: nex-stock-test.

This uses the embedded chat template. No private parser patch or custom chat template was necessary for the stock smoke checks. --load-mode none was used to avoid previously observed mapped-loading RAM pressure. This candidate has 40 trunk layers and no MTP tensors or MTP acceleration.

131,072 is the total context budget, including prompt, image tokens and generated output. It is not a promise of 131,072 input tokens plus unlimited output. Use one slot: VRAM headroom is small. Our machine used a 150 W power cap and an external 80°C guard; these protections are not installed by this command. Keep appropriate thermal and memory monitoring for your own system.

Quantization recipe

No training, model merging or expert pruning was performed. Calibration used Nex's tokenizer/template, 280 examples and 64,512 consumed tokens. Activation importance was collected using the Q5 reference; the quantizer input was BF16. See CALIBRATION-AND-QUANTIZATION.md for source revisions, importance-matrix hash, coverage limits and the quantizer command. The calibration matrix and raw calibration datasets are not distributed here; the recipe alone is insufficient for byte-identical requantization.

Measured coding results — regressions disclosed

These are our paired Python coding evaluations, not upstream Nex-AGI leaderboard scores. Q5KM is an approximate reference, not BF16.

HumanEval+ had 12 regressions and 10 improvements; MBPP+ had 29 regressions and 16 improvements. The no-measured-regression gate failed. Non-significant statistical tests do not establish equal quality.

The evaluator required ARM64/container/sharding recovery after memory failures. Saved responses were not regenerated to improve scores. See EVALUATION-METHOD.md for evaluator identity and the non-default recovery settings; they matter when interpreting these results.

Measured speed — earlier patched build, not stock b10896

RTX 5060 Ti 16 GB, Windows, 150 W, 131,072 allocated context, one slot, Q4_0 K/V, batch 512, microbatch 128. One warm-up plus three measured repetitions at each length; each measured request generated 128 tokens with zero cached prompt tokens. Values below are means on a synthetic workload.

The benchmark runtime reports commit 1208c5956237f8f2c8a767d1ad6f26d16d9eb0ee with local modifications. It is not the stock b10896 runtime above. Its exact historical build reproduction remains unverified, and its private patches and binaries are not distributed. These speeds are observations, not a promise that downloading stock llama.cpp will reproduce them. See REPRODUCTION.md, metrics.json, and EXPERIMENTAL-REPORT.md.

Vision, context and stability: successes and failures

• —Stock b10896: 128K allocation, exact text response, structured tool call and single-image OCR passed. The short run peaked at 42°C; sampled minimum free VRAM was 427 MiB, with no guard stop. Stock full-context inference and the full benchmark suite were not rerun.
• —Earlier runtime: cross-file arithmetic retrieval passed at four lengths, through 126,046 actual prompt tokens.
• —Strict multi-needle retrieval failed at all four lengths. At the three larger lengths the values were correct but JSON shape was wrong; the original failures remain failures. At 8K, the output limit was exhausted before a final answer.
• —A 126,412-token two-image request fit but failed accuracy, emitting no final answer within its 128-token output cap. The small multi-image fixture also failed. Do not treat this as strong general-purpose computer-use validation.
• —Custom GUI grounding scored only 2/6. The 90 synthetic fixtures scored 82/90 for the candidate and 49/90 for AMD Q5, but backend/parser differences confound that comparison; it does not establish superior model quality.
• —Thirty sequential requests, cancellation/follow-up and a controlled model restart passed on the earlier runtime. Deliberate oversized allocation was safety-blocked, not passed.
• —Earlier 180 W testing hit the 80°C guard and coincided with an NVIDIA driver event. Final 150 W long-context testing peaked at 76°C without a guard stop. Its sampled minimum free VRAM was only 245 MiB; shared-memory paging was not independently measured. Earlier kernel/driver incidents are not proven resolved.
• —Stock logs contain non-consecutive token-position warnings during image handling despite correct OCR. Broader image behavior remains unverified.

Small synthetic tests cannot establish broad multilingual coding, SWE-bench, BFCL, ScreenSpot or general agentic reliability. The multilingual custom coding cases contain Python instructions in four natural languages, not four programming languages. Large uncached prompts also require client/proxy timeouts compatible with long prefill latency.

Files, provenance and licensing

Original model revision: 87420286149d9cce9bd46cd335ef9bda33c37c1b. The upstream model card declares Apache-2.0. See LICENSE, NOTICE and MODIFICATIONS.txt.

This repository contains the tested GGUF, projector, a sanitized model card, quantization/runtime instructions and measured results. It excludes credentials, SSH configuration, private endpoint details, operational logs, raw datasets, private runtime patches and runtime binaries. Historical test failures are disclosed, not removed. Publication is an experimental release, not a claim that all validation gates passed.