webbrain-one/Laguna-XS-2.1-Vision-NVFP4

Laguna XS 2.1 Vision (NVFP4)

[image]

Laguna XS 2.1 with sight. A vision-language development checkpoint that connects Poolside's agentic coding model to the MoonViT vision encoder from Kimi-K2.6 through a trained PatchMerger projector.

The text backbone and vision tower remain frozen. The only newly trained parameters are the 30,679,808-parameter projector that merges each 2x2 group of MoonViT patches and maps the resulting 4608-dimensional representation into Laguna's 2048-dimensional token space.

Why vision at WebBrain

At WebBrain, we build browser agents that need to understand the visual state of the web—not just extracted text. Screenshots, charts, dashboards, rich editors, and the location and appearance of controls are part of real browser work, so vision is a practical product requirement.

In our American–Chinese open-model frontier benchmark, Laguna XS 2.1 stood out as a strong model in its size class, but the upstream checkpoint is text-only. This project adds a basic MoonViT vision bridge while keeping both the language backbone and vision tower frozen.

[!IMPORTANT] The pinned NVFP4 text backbone, the frozen MoonViT tower, and the final 100K-example projector are packaged and fingerprint-verified. Multimodal model code (modeling_laguna.py), a processor (image_processing_laguna.py / processing_laguna.py), and a dedicated vLLM serving path now exist — AutoModelForImageTextToText.from_pretrained(...) loads all three weight files and merges image embeddings into the text stream. NVFP4 equivalence and broader image-inference regression gates are still pending — see Experimental status below.

Build status

• —[x] Pin the upstream Laguna XS 2.1 NVFP4 revision.
• —[x] Copy and fingerprint-verify all 16 backbone files (21.60 GB).
• —[x] Pass backward and 2,048-token memory gates before the full run.
• —[x] Cache 100,000 MoonViT training examples.
• —[x] Finish the 100,000-example projector run.
• —[x] Add and fingerprint-verify the frozen MoonViT tower and final projector.
• —[x] Assemble multimodal configuration, processor, and serving integration.
• —[ ] Pass final single-GPU loading, image inference, and regression gates.

Included vision artifacts

Machine-readable provenance is in `VISION_ADAPTER_MANIFEST.json`.

Provenance

The packaged text backbone is copied from `poolside/Laguna-XS-2.1-NVFP4` at immutable revision `d32afde8b09af1539b49ff96ff5551c674485f8e`. Every copied file was checked against its upstream Git blob or LFS SHA-256 fingerprint, then independently rechecked after upload.

Projector training used the frozen BF16 Laguna XS 2.1 backbone and frozen MoonViT-3d features. This repository now pairs the final trained projector and frozen tower with the verified NVFP4 backbone above. NVFP4 equivalence and end-to-end behavior will be validated after the remaining multimodal integration is assembled.

Usage

transformers

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "webbrain-one/Laguna-XS-2.1-Vision-NVFP4"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda"
)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "https://example.com/screenshot.png"},
        {"type": "text", "text": "What does this screenshot show?"},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

from_pretrained resolves the NVFP4 text backbone through the normal model.safetensors.index.json path, then loads vision_tower.safetensors and mm_projector.safetensors directly onto model.vision_tower / model.multi_modal_projector — see LagunaForConditionalGeneration._load_vision_assets in modeling_laguna.py.

vLLM

There is a dedicated vLLM-native model (architecture laguna_vl, reusing vLLM's own MoonViT tower implementation instead of the plain-PyTorch one in modeling_laguna.py), but as a patch applied directly to a vLLM install — it is not part of any upstream vLLM release, so vllm serve webbrain-one/Laguna-XS-2.1-Vision-NVFP4 will not pick it up on a stock vLLM build. Whether vLLM's generic --trust-remote-code / Transformers-backend fallback can load this repo's custom multimodal code as-is has not been verified — see Experimental status below. Until either path is upstreamed and confirmed, transformers (above) is the supported way to run this checkpoint.

Method credit

The overall construction and model-card approach was inspired by Baseten's GLM-5.2-Vision-NVFP4: keep the text backbone and MoonViT tower frozen, train a compact PatchMerger projector between them, and publish the provenance and hardware constraints explicitly. Credit to the Baseten team for demonstrating this practical recipe. This repository does not reuse Baseten model weights, benchmark results, or deployment artifacts.

License

The redistributed Laguna XS 2.1 NVFP4 backbone remains subject to the included OpenMDW-1.1 license and Poolside's source notices. The included MoonViT tower remains subject to the included Kimi-K2.6 Modified MIT terms. The newly trained projector is documented in the manifest above. Downstream users remain responsible for complying with both upstream licenses.

Acknowledgements

Built on Poolside's Laguna XS 2.1 NVFP4 and Moonshot AI's Kimi-K2.6, with the vision-attachment method inspired by Baseten's GLM-5.2-Vision-NVFP4. These teams were not involved in this development checkpoint; please do not direct issues with this repository to them.

Want this model on your inference provider?

Ask your inference provider—such as OpenRouter or another OpenAI-compatible managed service—to deploy this exact repository with its multimodal processor and serving plugin. Deploying only the upstream text backbone will not enable image input.

Experimental status, roadmap, and get involved

[!WARNING] Experimental vision package. The MoonViT adapter and its serving package are experimental. Live end-to-end validation of the NVFP4 package on an RTX 5090 is still provisioning: the pinned SGLang container image is being fetched, so neither the text nor image smoke test has passed yet. Fine-grained OCR, small-object and control identification, GUI grounding, and hallucination calibration remain limited. The roadmap is to expand and diversify the training data, add more high-resolution OCR and UI examples, and pursue broader parameter-efficient tuning if community interest warrants the investment. To contribute evaluation or training data, sponsor compute, or explore a design partnership, use the community interest form. Do not use this package as the sole decision source for safety-critical automation.