poolside-laguna-hackathon/laguna-vision

Laguna Vision

Open-source GitHub · Hugging Face model

Laguna Vision adds a visual input path to poolside/Laguna-XS.2. SigLIP encodes images, AnyRes tiling preserves screenshot/document detail, a resampler projector maps features into Laguna's embedding space, and LoRA adapters are trained with supervised visual-instruction data.

Method: post-training multimodal adaptation via supervised fine-tuning.

Current status: latest is an early 200k-example checkpoint. It serves successfully but is weakly grounded: 12 / 80 strict passes on the live capability matrix.

[image]

Vision pathway breakdown

Laguna can generate text, but it has no native pixel input. This checkpoint adds the missing bridge from screen/image pixels into Laguna tokens.

Checkpoint

Capability matrix

The answer audit should live at evals/live_capability_eval_80/capability_probe.answers.rescored.jsonl.

What to keep in this model repo

Do not upload raw image archives, feature caches, access tokens, or full gated Laguna base weights.

Endpoint

Use the default Hugging Face Dedicated Inference Endpoint Python runtime with this repo's handler.py.

In the Hugging Face Inference Endpoint UI, paste one of these objects into the JSON body editor. A plain text payload such as {"inputs": "Hello world!"} is not enough; Laguna Vision needs inputs.image plus inputs.question.

Quick HF UI test payload:

{
  "inputs": {
    "image": "https://images.cocodataset.org/val2017/000000039769.jpg",
    "question": "What animals are in this image? Answer briefly.",
    "max_new_tokens": 64
  }
}

Same payload with curl:

HF_ENDPOINT=https://your-endpoint.endpoints.huggingface.cloud
HF_ENDPOINT_TOKEN=...

curl -s "${HF_ENDPOINT}" \
  -H "Authorization: Bearer ${HF_ENDPOINT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": {
      "image": "https://images.cocodataset.org/val2017/000000039769.jpg",
      "question": "What animals are in this image? Answer briefly.",
      "max_new_tokens": 64
    }
  }'

Generic request:

{
  "inputs": {
    "image": "https://example.com/image.jpg",
    "question": "What is shown in this image?",
    "max_new_tokens": 128
  }
}

Local image as a data URI:

IMAGE_DATA_URI="$(python3 - <<'PY'
import base64
from pathlib import Path

print("data:image/png;base64," + base64.b64encode(Path("path/to/image.png").read_bytes()).decode("ascii"))
PY
)"

curl -s "${HF_ENDPOINT}" \
  -H "Authorization: Bearer ${HF_ENDPOINT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{
    \"inputs\": {
      \"image\": \"${IMAGE_DATA_URI}\",
      \"question\": \"What is shown in this image?\",
      \"max_new_tokens\": 64
    }
  }"

OpenAI-style multimodal request:

{
  "inputs": {
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is shown?"},
          {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
        ]
      }
    ]
  }
}

Response:

{"answer": "...", "checkpoint": "latest"}

vLLM serving

For production vLLM serving, keep this HF endpoint as the reference path and run the repository gateway separately:

1. 1.Merge latest/lora into poolside/Laguna-XS.2 with laguna-vision-vllm merge-lora, or start vLLM with --enable-lora --lora-modules laguna-vision=latest/lora.
2. 2.Start vLLM with --trust-remote-code --enable-prompt-embeds.
3. 3.Start laguna-vision-vllm serve --checkpoint latest --vllm-base-url http://127.0.0.1:8000/v1 --model laguna-vision.
4. 4.Compare this endpoint and the vLLM gateway with laguna-vision-vllm compare-endpoints on evals/live_capability_eval_80/probe/manifest.jsonl.

The gateway sends a single full embedding tensor to vLLM's /v1/completions API using top-level prompt_embeds; it does not send prompt_embeds as a chat content part. Prime validation on 2026-05-30 confirmed this vLLM API works on vllm==0.10.2. The real poolside/Laguna-XS.2 backend needs an 80GB-class GPU or equivalent memory plan; a 1x A100 40GB pod reached the correct vLLM Transformers backend and then failed with CUDA OOM.

Limitations

• —Early checkpoint quality is uneven.
• —OCR, counting, charts, tables, and precise UI localization are unreliable.
• —The model can hallucinate when visual evidence is weak.
• —Validate outputs before using them in user-facing or high-stakes workflows.