hiyasvyas/ortho-vision-lora

Ortho-Vision LoRA — third-angle orthographic views from a 3D block image

A QLoRA adapter for Qwen2.5-VL-3B-Instruct that, given an image of a solid built from unit cubes, outputs its third-angle orthographic projections (top / front / right) as three ASCII grids. One narrow, reliable behavior instilled by fine-tuning — not prompting.

• —Base model: unsloth/Qwen2.5-VL-3B-Instruct-bnb-4bit
• —Adapter type: LoRA (r=16, alpha=16), ~41M trainable params (1.08% of the model)
• —Training data: `hiyasvyas/orthographic-views-vision` — 1,500 train / 150 val, deterministic labels
• —Training: 2 epochs, lr 2e-4, batch 2 x grad-accum 4, Unsloth QLoRA on a Colab T4

Behavior Spec (the litmus test)

Given an image of a 3D unit-cube solid, output exactly three ASCII grids labeled \top:\, \front:\, \right:\ — using only \.\ and \#\, with no prose before, between, or after the grids.

A well-prompted base VLM cannot do this reliably; it never even produces the three-grid format. Fine-tuning fixes the format completely and improves the spatial content.

Results — base vs. tuned (100 held-out golden images, exact-match)

Read: the base model scores zero everywhere — it cannot produce the format at all. The tuned model produces the correct 3-grid format 100% of the time and gets individual views right a meaningful fraction of the time. Format adherence (the target behavior) is fully instilled; per-view spatial accuracy is the next data-iteration target.

How to use

\\\`python from transformers import AutoProcessor, Qwen25VLForConditionalGeneration from peft import PeftModel from PIL import Image import torch

base = "unsloth/Qwen2.5-VL-3B-Instruct-bnb-4bit" model = Qwen25VLForConditionalGeneration.frompretrained(base, torchdtype=torch.float16, devicemap="auto") model = PeftModel.frompretrained(model, "hiyasvyas/ortho-vision-lora") processor = AutoProcessor.from_pretrained("hiyasvyas/ortho-vision-lora")

prompt = ("Look at this 3D object built from unit cubes. Output its third-angle " "orthographic views as three ASCII grids labeled 'top:', 'front:', and " "'right:', using only '.' and '#'. Output nothing except the three grids.") img = Image.open("yourblockimage.png").convert("RGB") msgs = [{"role": "user", "content": [{"type": "image", "image": img}, {"type": "text", "text": prompt}]}] inputs = processor.applychattemplate(msgs, addgenerationprompt=True, tokenize=True, returndict=True, returntensors="pt").to(model.device) out = model.generate(**inputs, maxnewtokens=256) print(processor.decode(out[0], skipspecialtokens=True)) \\\`

Limitations

Per-view spatial accuracy is still low (\exact\ = 3%). \top_ok\ (7%) lags \front\/\right\, suggesting the top-down footprint is hardest to read from the rendered perspective — a data/rendering signal to fix in the next iteration, not a hyperparameter problem.

Framework versions

• —PEFT 0.19.1