rodrigoramosrs/veriloop-coder-e2-nvfp4

1---2library_name: transformers3pipeline_tag: text-generation4license: apache-2.05base_model:6  - tsinghua-sigs-robot-lab/VeriLoop-E27base_model_relation: quantized8language:9  - en10  - zh11tags:12  - veriloop13  - veriloop-coder14  - code15  - coding-agent16  - software-engineering17  - mathematical-reasoning18  - nvfp419  - modelopt20  - tensorrt-llm21  - vllm22  - sglang23  - code-optimized24  - quantization25  - open-source26  - apache-2.027  - qwen3_528  - self-harness29  - harness-engineering30  - surface-host-adapter31  - evidence-binding32  - rollback33  - uncertainty-calibration34  - long-context35  - vertical-code-model36  - recursive-improvement37---38 39<div align="center">40  <h1>VeriLoop Coder E2 · NVFP4</h1>41  <p><strong>Coding-Optimized Quantized Model (NVIDIA NVFP4)</strong></p>42  <p>43    <a href="https://huggingface.co/tsinghua-sigs-robot-lab/VeriLoop-E2">Original Model ↗</a>44    ·45    <a href="https://github.com/rodrigoramosrs">GitHub</a>46    ·47    Apache-2.048  </p>49</div>50 51---52 53## Overview54 55This repository contains an **NVFP4 quantization** of [VeriLoop E2](https://huggingface.co/tsinghua-sigs-robot-lab/VeriLoop-E2), an open 27B post-trained model built on **Qwen3.8-27B** for code, mathematics, and physics. Its core reasoning discipline is **VeriLoop-Governed Recurrence (VGR)**: candidate states are recursively proposed, externally checked, and retained only when the protected evidence state improves without regression.56 57Quantized by [Rodrigo Ramos](https://github.com/rodrigoramosrs).58 59## Quantization Approach60 61Produced with [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer) using the canonical **`NVFP4_W4A4_WEIGHT_LOCAL_HESSIAN_CFG`** recipe, verbatim: static per-block 4-bit weights (group size 16) + dynamic 4-bit activations, FP8 attention, local-Hessian calibration with MSE and fp8 scale sweep.62 63Calibration ran on **512 samples × 512 tokens** (262144 tokens) of competitive-programming code from `nvidia/Nemotron-Competitive-Programming-v1`, keeping quantization faithful on the distributions that matter most for coding tasks.64 65The result retains the original model's strong software-engineering capabilities at roughly one third of the BF16 footprint, ready for NVFP4-capable inference stacks on Blackwell GPUs.66 67## Files68 69| File | Size | Notes |70|---|---|---|71| `model-00001-of-00005.safetensors` … `model-00005-of-00005.safetensors` | ~17.5 GB total | NVFP4 weights (packed U8 + FP8 block scales) |72| `model.safetensors.index.json` | — | Shard index (2051 tensors) |73| `hf_quant_config.json` | — | `quant_algo: NVFP4`, group size 16 |74| `config.json`, `tokenizer.*`, `merges.txt`, `vocab.json` | — | Config + tokenizer from the base model |75| `scripts/quantize_veriloop.py` | — | Full reproduction script (multi-GPU PTQ) |76| `scripts/validate_nvfp4.py` | — | Structural shard/index validator |77 78## Usage79 80This checkpoint targets **NVFP4-capable runtimes** (Blackwell GPUs). Plain `transformers` cannot load modelopt NVFP4 checkpoints directly.81 82> **Compatibility note:** per the NVIDIA recipe, a small set of projections (`linear_attn.in_proj_a/b`, `conv1d`), plus the embeddings and `lm_head`, intentionally ship in BF16. The `exclude_modules` list in `hf_quant_config.json` (mirrored in `config.json`) declares exactly which modules those are — inference loaders must honor it and serve those modules unquantized. If a loader errors with *"weight is torch.bfloat16 but the quant config declares NVFP4"*, it is ignoring the exclusion list, not a checkpoint defect.83 84### vLLM85 86```bash87pip install 'vllm>=0.17.0'88 89vllm serve rodrigoramosrs/veriloop-coder-e2-nvfp4 \90    --tensor-parallel-size 1 \91    --max-model-len 131072 \92    --kv-cache-dtype fp8_e4m3 \93    --gpu-memory-utilization 0.9294```95 96### SGLang97 98```bash99pip install 'sglang[all]'100 101python -m sglang.launch_server \102    --model-path rodrigoramosrs/veriloop-coder-e2-nvfp4 \103    --trust-remote-code \104    --mem-fraction-static 0.88105```106 107### TensorRT-LLM108 109Build the engine from this checkpoint with `trtllm-build` using an NVFP4 + FP8-KV configuration, then serve with `trtllm-serve`. See the [TensorRT-LLM documentation](https://github.com/NVIDIA/TensorRT-LLM) for the exact flags for your GPU.110 111### Reproducing112 113```bash114pip install "nvidia-modelopt[all]" torch transformers accelerate datasets safetensors115 116# needs ~55 GB VRAM across CUDA GPUs for the 27B BF16 source model117python scripts/quantize_veriloop.py \118    --model ./model-bf16 \119    --output ./model-nvfp4 \120    --calib-size 512 \121    --calib-seq-len 512122 123python scripts/validate_nvfp4.py ./model-nvfp4124```125 126On Windows the Triton JIT needs a C compiler: run from a Visual Studio Native Tools prompt (`vcvars64`) with `CC` pointing at `cl.exe`. See the header of `scripts/quantize_veriloop.py` for multi-GPU layout tuning (`--layers-split`, `--gpu-order`).127 128## Acknowledgements129 130- **Libo Wang** and the **Intelligent Robotics Laboratory, Tsinghua SIGS** for developing the original VeriLoop E2 model.131- **NVIDIA** for Model Optimizer and the NVFP4 quantization tooling.132- The original model repository: [tsinghua-sigs-robot-lab/VeriLoop-E2](https://huggingface.co/tsinghua-sigs-robot-lab/VeriLoop-E2)133 134## License135 136Apache-2.0. The weights are quantized from the original Apache-2.0 licensed model. See the [original repository](https://huggingface.co/tsinghua-sigs-robot-lab/VeriLoop-E2) for full licensing details and third-party notices.137