hiyasvyas/orthographic-views-vision

Orthographic Projection Views — Vision

Fine-tuning data for a small vision-language model that, given a rendered image of a 3D block object, outputs its third-angle orthographic projections (top / front / right) as three ASCII grids. Built to instill one narrow, reliable behavior via QLoRA on a small open VLM (Qwen2.5-VL-3B class).

Behavior Spec (the litmus test)

Given an image of a solid built from unit cubes, output exactly three ASCII grids labeled top:, front:, right: — top = Y×X footprint, front = Z×X, right = Z×Y (third-angle) — using only . and #, with no prose before, between, or after the grids.

A well-prompted small base VLM cannot do this reliably: it drifts on view alignment, grid dimensions, and hidden/occluded structure. That unreliability is exactly what fine-tuning fixes — and what makes this a valid target rather than something to just prompt.

Why the labels are trustworthy

The three views are a deterministic function of the underlying voxel solid, so every label is computed programmatically (project()), not sampled from a teacher model. Labels are therefore 100% correct by construction — no distillation noise — and the eval is exact-match, not a fuzzy judge. The model reads pixels and emits text; it never generates images.

Configs & splits

The golden config is held out from training and carries extra fields (id, source, difficulty, dims, tags) for stratified error analysis.

Record format

{
  "image": "<PNG, embedded>",
  "prompt": "Look at this 3D object built from unit cubes. Output its third-angle orthographic views ...",
  "completion": "top:\n##..\n###.\nfront:\n###.\n##..\nright:\n###\n###"
}

Conventions (baked into the labels)

• —X width (left→right), Y depth (front y=0→back), Z height (bottom z=0→top).
• —top = Y×X footprint, front edge at the bottom row.
• —front = Z×X, object top is the first row.
• —right = Z×Y, object top first, front edge in the left column.

Provenance

• —Real shapes: occupancy grids from Objaverse (via PeterAM4/blockgen-3d), cropped to their bounding box and downsampled to small grids, then rendered and projected.
• —Synthetic shapes: random contiguous / extruded solids for difficulty coverage. Fully-solid and empty grids are rejected (they project trivially).

Evaluation

Score format_ok, top_ok, front_ok, right_ok, and exact (all three views correct) against the deterministic ground truth on the golden config. The fine-tune must beat the base model on exact-match to count as a win.

Loading

from datasets import load_dataset

train = load_dataset("hiyasvyas/orthographic-views-vision", "default", split="train")
golden = load_dataset("hiyasvyas/orthographic-views-vision", "golden", split="golden")
img = train[0]["image"]          # PIL.Image
print(train[0]["completion"])    # the three ASCII grids