coder543/North-Mini-Code-1.0-QAD-GGUF

1---2inference: false3library_name: transformers4license: apache-2.05tags:6- conversational7- chat8- code9- agent10base_model: CohereLabs/North-Mini-Code-1.0-w4a1611---12 13## GGUF Conversion Note14 15This repository contains a GGUF conversion of `CohereLabs/North-Mini-Code-1.0-w4a16`, Cohere's QAD-trained NVFP4 W4A16 checkpoint for North Mini Code. The GGUF was produced with `llama.cpp`'s `convert_hf_to_gguf.py` using `--outtype bf16`.16 17The expert weights were repacked from the source checkpoint's `compressed-tensors` `nvfp4-pack-quantized` format into GGUF `NVFP4` tensors. They were not dequantized and requantized into a standard GGUF `Q4_*` or `IQ4_*` format, so this conversion is intended to preserve the QAD-trained 4-bit weights. Non-NVFP4 tensors are stored as BF16 or F32 as emitted by the converter.18 19During conversion, the local `llama.cpp` converter needed a small workaround for a lazy tensor zero-bias check in the Cohere2 MoE path; the checkpoint's bias tensors were zero and were skipped as intended.20 21Conversion command:22 23```shell24python convert_hf_to_gguf.py \25  /path/to/CohereLabs/North-Mini-Code-1.0-w4a16 \26  --outfile north-mini-code-1.0-w4a16-nvfp4.gguf \27  --outtype bf1628```29 30# **Model Card for North Mini Code**31 32## **Model Summary**33 34North Mini Code is an open weights research release of a 30B-A3B parameter model optimized for code generation, agentic software engineering, and terminal tasks.35 36Developed by: [Cohere](https://cohere.com/) and [Cohere Labs](https://cohere.com/research)37 38* Point of Contact: [**Cohere Labs**](https://cohere.com/research)39* License: Apache 2.040* Model: North Mini Code41* Model Size: 30B total; 3B active42* Context length: 256K & 64K max output43* Quantization: NVFP4 W4A1644 45For more details about this model, please check out our [blog post](https://huggingface.co/blog/CohereLabs/introducing-north-mini-code).46 47**Try North Mini Code**48 49You can try out North Mini Code before downloading the weights in OpenCode and our hosted [Hugging Face Space](https://huggingface.co/spaces/CohereLabs/North-Mini-Code-1.0).50 51**Evaluation**52 53![image1](https://cdn-uploads.huggingface.co/production/uploads/62668f725fb8d521d94d8451/xR7kZ3X9RKEZrbgD6hpG1.png)54 55<details>56<summary><span style="font-size: 80%;"><b>Benchmarking Methodology [CLICK TO EXPAND]</span></b></summary>57 58- <span style="font-size: 80%;">We used SWE-Bench Verified, SWE-Bench Pro, Terminal-Bench v2, and Terminal-Bench Hard to benchmark North Mini Code's agentic coding capabilities. For evaluation harnesses, we used the Swe-Agent harness v1.1.0 for SWE-Bench, and a simple ReAct harness employing a single terminal-use tool based on Harbor's Tmux session implementation for Terminal-Bench v2. For Terminal Bench Hard, we directly used Terminus-2, following the same methodology as the Artificial Analysis Intelligence Index to compare North-Mini-Code-1.0 with the other models. Additionally, we used SciCode and LiveCodeBench v6 as complex code-generation benchmarks outside of tool use.</span>59- <span style="font-size: 80%;">We run each benchmark with 3 different seeds and report the average benchmark performance, using temperature=1.0 and top\_p=0.95. We used publicly reported scores for competitor models, either from original reports or the Artificial Analysis Intelligence Index, where available. Additionally, Gemma4’s scores for agentic coding tasks were reported by [Qwen team](https://qwen.ai/blog?id=qwen3.6-35b-a3b). For benchmark results that any public report is missing, denoted by (\*) in the figure, we run them internally using the recommended model configuration.</span>60</details>61 62**Usage**63 64To use our model in transformers, please use our BF16 model weights. Our NVFP4\_W4A16 checkpoint is designed to be used with vLLM and MLX-VLM and is not compatible with transformers due to lack of native 4-bit support.65 66### **vLLM**67 68You can run the model in vLLM. Please use vLLM main for North Mini Code until a new release is available, and accurate response parsing also requires installing Cohere’s melody library.69 70```shell71uv pip install "git+https://github.com/vllm-project/vllm.git"72uv pip install cohere_melody>=0.9.073```74 75Then the vLLM server can be started with the following command:76 77```shell78vllm serve CohereLabs/North-Mini-Code-1.0-w4a16 \79  -tp 1 \80  --max-model-len 320000 \81  --tool-call-parser cohere_command4 \82  --reasoning-parser cohere_command4 \83  --enable-auto-tool-choice84```85 86**Use locally deployed North Mini Code in OpenCode:**87 88Please use OpenCode \> v1.17.0.89 90```shell91brew install anomalyco/tap/opencode92```93 94To use locally deployed North Mini Code in Opencode, please use this config which enables interleaved reasoning:95 96```json97{98  "$schema": "https://opencode.ai/config.json",99  "model": "vllm/CohereLabs/North-Mini-Code-1.0-w4a16",100  "provider": {101    "vllm": {102      "npm": "@ai-sdk/openai-compatible",103      "name": "Local vLLM server",104      "options": {105        "baseURL": "http://127.0.0.1:8000/v1",106        "apiKey": "EMPTY"107      },108      "models": {109        "CohereLabs/North-Mini-Code-1.0-w4a16": {110          "name": "North-Mini-Code-1.0",111          "interleaved": {112            "field": "reasoning"113          },114          "limit": {115            "context": 256000,116            "output": 64000117          }118        }119      }120    }121  }122}123 124```125 126### **MLX-VLM**127 128You can also run the model in MLX-VLM. Please use main for North Mini Code until a new release is available.129 130```shell131uv pip install "git+https://github.com/Blaizzy/mlx-vlm.git@main"132```133 134Then the mlx\_vlm server can be started with the following command:135 136```shell137mlx_vlm.server \138  --model CohereLabs/North-Mini-Code-1.0-w4a16 \139  --enable-thinking \140  --thinking-start-token "<|START_THINKING|>" \141  --thinking-end-token "<|END_THINKING|>"142```143 144**Opencode config:**145 146Actual limit depends on your device147 148```json149{150  "$schema": "https://opencode.ai/config.json",151  "model": "mlx-vlm/CohereLabs/North-Mini-Code-1.0-w4a16",152  "provider": {153    "mlx-vlm": {154      "npm": "@ai-sdk/openai-compatible",155      "name": "MLX VLM Local",156      "options": {157        "baseURL": "http://127.0.0.1:8080/v1",158        "apiKey": "EMPTY"159      },160      "models": {161        "CohereLabs/North-Mini-Code-1.0-w4a16": {162          "name": "North-Mini-Code-1.0",163          "interleaved": {164            "field": "reasoning"165          },166          "limit": {167            "context": 256000,168            "output": 64000169          }170        }171      }172    }173  }174}175```176 177## **Model Details**178 179**Input**: Text only.180 181**Output**: Model generates text.182 183**Model Architecture**: North-Mini-Code-1.0 is a decoder-only Transformer-based sparse Mixture-of-Experts model. It uses an efficient attention implementation, interleaved between sliding-window attention with RoPE and global attention with no positional embeddings, in a 3:1 ratio. The feed-forward block is an MoE block with 128 experts, of which 8 are activated per token. Each expert block is an FFN block with SwiGLU activation. The router applies a sigmoid activation function to the logits before the top-k selection. We also use a single dense layer before the sparse layers. North-Mini-Code-1.0 was post-trained using a two-stage cascaded supervised fine-tuning (SFT) followed by reinforcement learning with verifiable rewards (RLVR), focusing on agentic coding.  For more technical details, please check out our [blog post](https://huggingface.co/blog/CohereLabs/introducing-north-mini-code).184 185**Quantization Methodology:** We use NVFP4 W4A16 quantization (4-bit weights, 16-bit activations) for this model, delivering a much smaller memory footprint (\~ 18-20GB) and faster inference while preserving coding accuracy. We quantize the MoE experts only, keeping attention, the dense layer, and the router at higher precision. Since the experts hold most of the model's parameters, this captures the bulk of the savings with minimal quality loss. To preserve quality, we use Quantization-Aware Distillation (QAD), training the quantized model to match the unquantized model's outputs, **achieving \>99% overall accuracy recovery** across our evaluations. Since only weights are quantized, this format does not require native FP4 hardware and runs on pre-Blackwell GPUs such as Hopper and Ada.186 187**Context Length:** North-Mini-Code-1.0 supports a context length of 256K & 64K output length.188 189## **Model Card Contact**190 191For errors or additional questions about details in this model card, contact \[labs@cohere.com\].192