BDivyesh/boomi-stage3-vlm-cubicasa-research

BOOMI — Stage-3 Raster→SPEC VLM (Qwen2.5-VL-3B LoRA) · RESEARCH-ONLY

⚠️ NON-COMMERCIAL / RESEARCH-ONLY. This LoRA adapter was fine-tuned on CubiCasa5K (CC-BY-NC-SA 4.0). It is released under CC-BY-NC-SA 4.0 and must not be used commercially. It is not part of BOOMI's license-clean shipped model line (those are the BDivyesh/boomi-stage-a-* text→SPEC models, trained only on permissive data). Use it for research and benchmarking only, with attribution to CubiCasa5K.

A LoRA adapter for `Qwen/Qwen2.5-VL-3B-Instruct` (Apache-2.0 base) that reads a residential floor-plan image and emits a structured room program as JSON:

{"room_program": {"BED ROOM": 3, "BATH": 2, "KITCHEN": 1, "LIVING": 1, "BALCONY": 2, "FOYER": 1, "STORE": 1, "LOBBY": 1}, "bhk": "3BHK"}

This is the raster modality of BOOMI's reverse pipeline (arbitrary drawing → structured SPEC), for scanned/photo plans where vector geometry and in-drawing text are unavailable. The SPEC it produces is intended to seed a downstream constraint solver — it is an assistive proposer a human reviews and corrects, not an autonomous extractor.

Intended use

• —In scope: research on raster floor-plan understanding; producing a draft room program from a plan image that a user then edits; benchmarking against CubiCasa5K.
• —Out of scope: any commercial use (license); autonomous/unsupervised extraction (exact whole-program accuracy is low — see below); non-CubiCasa-style or non-Finnish plans without re-validation (transfer is unmeasured and expected to be lower).

Results (CubiCasa5K test split, n=149, fair multiset scorer)

Scored vs the CubiCasa SVG room-class annotations (independent of the model's pixels), with room-type names canonicalised to a shared label space (synonyms folded on both sides).

Read this honestly: fine-tuning materially improves which room types are present and the rough bedroom count, but exact whole-program match is only ~3.4 % — counts of balconies, baths and stores routinely drift by ±1, and the model leans on learned program priors (a few templates cover a large share of outputs). It is useful as a draft generator, not a source of truth.

How to use

import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from PIL import Image

BASE = "Qwen/Qwen2.5-VL-3B-Instruct"
ADAPTER = "BDivyesh/boomi-stage3-vlm-cubicasa-research"

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, use_safetensors=True).to("cuda")
model = PeftModel.from_pretrained(model, ADAPTER).to("cuda").eval()
proc = AutoProcessor.from_pretrained(BASE, max_pixels=1280*28*28)

PROMPT = ('You are an expert architect reading a residential floor-plan image. '
          'Identify every distinct room and return ONLY JSON: '
          '{"room_program":{"<TYPE>":<count>,...},"bhk":"<n>BHK"}. '
          'bhk = number of bedrooms. Count each room instance. Return only the JSON.')

img = Image.open("plan.png").convert("RGB")
msgs = [{"role": "user", "content": [{"type": "image", "image": "plan.png"},
                                     {"type": "text", "text": PROMPT}]}]
text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inp = proc(text=[text], images=[img], return_tensors="pt").to("cuda")
out = model.generate(**inp, max_new_tokens=512, do_sample=False)
print(proc.batch_decode(out[:, inp.input_ids.shape[1]:], skip_special_tokens=True)[0])

Training

• —Base: Qwen/Qwen2.5-VL-3B-Instruct (Apache-2.0).
• —Method: LoRA (r=16, α=32, dropout=0.05) on the LLM attention + MLP projections only; the vision tower is frozen (to limit memorising CubiCasa's render style). ~0.98 % of params trainable.
• —Data: CubiCasa5K train split; targets are the SVG-derived room program + BHK. Plans with empty/undefined programs were skipped. Frozen doc-level train/test split (no overlap).
• —Schedule: 1 epoch, lr 1e-4 cosine, bf16, gradient checkpointing, DDP on 2× H100; train loss 0.66 → 0.30.

Limitations & biases

• —Non-commercial / CubiCasa-derived (license).
• —Finnish-domain: CubiCasa is Finnish residential; the base zero-shot even leaks Finnish in-drawing labels (e.g. MH, ET). Indian / other-region transfer is unvalidated.
• —Assistive, not autonomous: low exact-program accuracy; relies on learned priors.
• —Counts are approximate; room types present are more reliable than their exact counts.

License & attribution

• —This adapter: CC-BY-NC-SA 4.0 (non-commercial, share-alike, attribution).
• —Base model Qwen/Qwen2.5-VL-3B-Instruct: Apache-2.0 (its own license applies).
• —Training data CubiCasa5K: CC-BY-NC-SA 4.0 — please cite/attribute CubiCasa5K if you use this adapter.

Framework versions

• —PEFT 0.19.1