yrlyrl/spatial-mmcot-vdrop

Spatial MMCoT v1 · vdrop

VDrop cross-view (QianYangMILA/vdrop-crossview-8k, arXiv:2605.27310), Infinigen Indoors rooms. Two egocentric views of one room with partly overlapping fields of view are the input (input_image_0 = cam0, input_image_1 = cam1); the target is upstream's ground-truth panorama render, a wide panorama of the room that spans both views. The questions are multiple choice and the trained answer is the option letter. The upstream release ships no reasoning text at all, so every thought here is EMPTY: output_text_list is exactly <think></think><image_start> and <image_end><think></think><answer>X</answer>. The panorama belongs to the scene, not the question: every question about a scene shares that scene's two input views and its one panorama, so the 7,015 rows use 1,158 distinct panoramas. Report validation scores with confidence intervals clustered by scene (for example a bootstrap over meta.scene_id), not as independent rows. The scenes were generated with Infinigen Indoors (github.com/princeton-vl/infinigen, whose code is BSD-3-Clause); the renders are distributed by the VDrop authors under CC-BY-4.0. Changes from upstream: the PNG images are re-encoded as JPEG and downscaled (the two input views from 1280x720 to 512x288, the panorama to 1,024 px wide); of the upstream thinking images only the panorama is kept; perspective-taking questions phrased relative to 'your partner' are dropped (as the upstream authors' build_interleaved_from_visual_only.py does by default); rows are removed by answer-prior balancing and split into train/validation by scene (both below); the ThinkMorph system prompt is prepended to the unchanged question.

Supervision kind (supervision_kind in meta): visual_only on every row: the thoughts are empty: the rows supervise the image and the answer only.

Upstream: `QianYangMILA/vdrop-crossview-8k`. Licence: cc-by-4.0.

Known issues

No row of this source is listed in reports/known_issues/ (its summary.json is empty); the measured caveats below still apply.

Measured caveats

Measured on this release by the pre-publication review (2026-09-25): problems that cannot be listed row by row (a shortcut in the options, a label convention, an upstream labelling scheme) and what the review found around the lists above. Where a caveat counts listed rows ("listed as ..."), the count is the table's, read from reports/known_issues/summary.json. Its other numbers are the review's own measurements, which no file carries: they hold for exactly these rows and are not re-measured automatically. Items marked Training-signal defect are problems in what the rows teach, not only in how they are described; no row was removed for them.

• —Validation is strongly clustered by scene. Its 267 rows come from 43 scenes: one scene supplies 52 of them, and the three largest supply 113 (42.3%). In training, 1,115 scenes give a median of 3 rows per scene and at most 100 (one scene's 100 rows share one panorama). Report validation scores with confidence intervals clustered by scene, not as 267 independent rows.

Size

A slot is one target position in one row; rows can share a target image (same scene or same intermediate state), and a row can repeat one of its own targets, so there are fewer distinct images (by content hash) than slots.

Input images per row: 2. Target images per row (the images the model is trained to generate): 1. Image corpus (source_scene_corpus): infinigen 7,015.

Row format

One row is: input image(s) and a question, then K rounds of thought → target image (the target is the source's own ground-truth image, which the model is trained to generate), then a final thought (normally a read-back of the last target; where a source's final thought is something else, or often leaves out the answer, the source note or Known issues says so) and the answer; here K is 1. In the train config:

image_list        list<binary>  inputs first, then the K target images in order
num_input_images  int64         how many of image_list are inputs
instruction_list  list<string>  one element: system prompt + question + options
output_text_list  list<string>  K+1 elements:
  [0]   <think>plan 1</think><image_start>
  [j]   <image_end><think>plan j+1</think><image_start>
  [K]   <image_end><think>read-back</think><answer>answer</answer>
row_uid           string        join key to `meta` and `preview`

Every image is a JPEG, and no input image is larger than 512 px on its long edge (measured on this release, 2026-09-25); the size each target was stored at is target_px in meta.

On every row <answer> holds the option key (answer type mcq_letter 7,015) that the model is trained to emit (a letter for mcq_letter), while meta.answer_value (the answer column of preview) holds that option's text. Map the key through the options listed in the question before comparing the two, and score model output against <answer>.

The system prompt is ThinkMorph's VLM_THINK_SYSTEM_PROMPT from its inferencer.py, verbatim (GEN_THINK_SYSTEM_PROMPT there has the same text), including its leading and trailing newline. The markers are plain strings, not tokenizer special tokens; the prompt writes </image_end> and the data writes <image_end>, exactly as the ThinkMorph-7B checkpoint was trained.

preview shows the same rows with one column per slot: input_image_i for the inputs; for each of the K = num_steps rounds, the plan thought_j and its target target_image_j; and the read-back in thought_1 on every row.

meta holds the per-row sidecar: task, scene_id and geometry_uid (the scene and geometry keys; the split key is named in the split paragraph below), trajectory_id (a camera-path or sample label, empty where the source has none), num_steps, num_input_images, answer_type, answer_value, majority_class_rate, target_image_kind, target_px, est_tokens, licence, split (train / validation, the Hub split names), supervision_kind (full_interleaved / visual_aux / visual_only) and filter_flags. majority_class_rate is the share of the task's most frequent answer_value among its training rows: it measures answer skew and is not a guessing baseline (where a task mixes question types or each row has its own options it can be far below chance); compare scores with the text-only baselines below.

Per-row license in meta: cc-by-4.0 7,015.

Flags on released rows (filter_flags in meta and preview, comma-separated):

Training with a BAGEL-family loader

Rows here have 2 input images: the first num_input_images entries of image_list are inputs and the rest are targets, so the loader must read num_input_images. The UnifiedEditIterableDataset of the IPT release (https://github.com/weikaih04/Imaginative-Perception-Token, data/interleave_datasets/edit_dataset.py) does: its parse_row conditions on image_list[:num_input_images] and trains the remaining images as targets, one after each output_text_list element but the last. The stock ThinkMorph loader (the same class in https://github.com/ThinkMorph/ThinkMorph) does not: it conditions on image_list[0] only and trains image_list[j+1] after output_text_list[j], the answer element included, so every input after the first is trained as a generated image and every target moves one slot later per extra input. Every row here has one target, so on a row with exactly two inputs it trains image_list[1] (the second input view) after the plan and image_list[2] (the real target) after the answer. It raises no error. To use it, make two changes in its parse_row:

k = int(row.get("num_input_images", 1) or 1)
for im in images[:k]:          # replaces the single _add_image(images[0], ...)
    data = self._add_image(data, pil_img2rgb(Image.open(io.BytesIO(im))),
                           need_loss=False, need_vae=True, need_vit=True)
...
img_idx = idx + k               # replaces img_idx = idx + 1

The stock BAGEL edit loader (ByteDance-Seed/Bagel) cannot train these rows: it never reads output_text_list and expects each instruction_list element to be a list of paraphrases.

parquet_info.json keys each training chunk as <source>/<split>/<file>, here vdrop/train/chunk_00000.parquet, with row-group counts read from the parquet footers. The loader matches a chunk only when its key equals the path it builds, os.path.join(data_dir, file), and skips a chunk with no key without a warning: a source that is alone in its group then fails with IndexError: list index out of range, and in a mixed group it adds no rows. Download into a directory named after the source, not after the repository:

from huggingface_hub import snapshot_download
snapshot_download("yrlyrl/spatial-mmcot-vdrop", repo_type="dataset", local_dir="<root>/vdrop",
                  allow_patterns=["train/*", "validation/*", "parquet_info.json", "reports/known_issues/*"])

Then either run from <root> with data_dir: vdrop/train and parquet_info_path: vdrop/parquet_info.json, or rebuild the index with absolute keys and use an absolute data_dir:

import json, os
root = "/abs/path/to/root"                      # the directory that holds vdrop/
info = json.load(open(os.path.join(root, "vdrop", "parquet_info.json")))
info = {os.path.join(root, k): v for k, v in info.items()}
json.dump(info, open(os.path.join(root, "vdrop", "parquet_info_abs.json"), "w"))
# data_dir = os.path.join(root, "vdrop", "train")   (spelled exactly so, no trailing slash)
# parquet_info_path = os.path.join(root, "vdrop", "parquet_info_abs.json")

The Hugging Face cache (.../snapshots/<hash>/train/) or a folder named spatial-mmcot-vdrop matches no key.

num_used_data counts chunk files, not rows: the loader repeats this source's file list up to that number, lists every (file, row group) pair, and deals whole row groups out, floor(R / worldsize) to each rank and floor(that / numworkers) to each DataLoader worker. The remainder is never read. This source has 1 training chunk file holding 53 row groups of up to 128 rows, so keep num_used_data large, e.g. the 128 of ThinkMorph's interleaved_reasoning.yaml (upstream's example.yaml asks for more than GPUs x workers); every row group is then read. Set to 1 and alone in its group on 8 GPUs with 4 workers, it reads only 32 of the 53 row groups. In a run that mixes sources, give each source the same multiple of its own training chunk-file count, e.g. 128 per file (128 here): the file list is repeated up to num_used_data entries, so a flat 128 for every source would read a two-file source's rows half as often as a one-file source's.

How the rows were chosen

The rows refused before conversion are perspective-taking questions asked relative to "your partner's perspective": most name no image, so the partner's viewpoint cannot be recovered from the two input views, and the rest are dropped too, as the upstream authors' own build script does.

Every removed row has one line, with its reason, in reports/:

Every line of s13_dropped.jsonl has reason: prior_downsample; step names the balancing pass that removed it, and split is written train or val (the Hub's validation).

S0raw lines in build/dropped.jsonl were refused before a release row existed, so their row_uid field holds the converter's key for the upstream record (the upstream qa config's id, e.g. perspective_taking_000002), not a 16-hex row_uid; lines from later steps carry the row_uid the row had. No removed row appears in meta or preview.

<details><summary>Per-step counters of the conversion</summary>

7,728 upstream rows were read; S0raw refused 158 before a row existed and passed 7,570 to the first step. S0 runs once more, last, on the final bytes. The reason for every refused, dropped or quarantined row is in the files above.

</details>

The train/validation split keeps rows sharing a scene_id in meta on one side, and the assignment is frozen (splits/ in the summary repository). S12 saw 7,570 rows under 1,251 keys. No validation input image has the content of a training input image, and none is a pixel-level near-copy of one. S12 does not record per source whether that test ran, but it skips it only for a source whose spec sets split_leak_pixels: false, and no spec does; over all sources it compared 21,661 candidate pairs (perceptual hash within 6 bits) pixel by pixel and found no near-copy (checked 2026-09-25).

Answer-prior balancing (S13)

Each (task, split) group is checked separately. An answer is the answer value compared as lower-cased text without a trailing full stop, with 'farther' read as 'further' and 'nearer' as 'closer' (for multiple choice, the option text, not the letter; where the candidates are drawn in the image, as in zebrajigsaw and zebratetris, the answer is the letter itself). An answer is real when it holds at least 5 rows and 2% of the group; k is the number of real answers. Answer step: the target is max(30%, 1/k) when k >= 2, and max(30%, 1/d) over the d distinct answers when k = 1; a validation group uses the larger of its own target and its task's train target. A group is cut only when k >= 1 and its most common answer holds more than the target plus 5 percentage points; every answer is then capped at one common count, chosen so that none exceeds the target, and smaller answers keep all their rows. At the answer step, a group at or below that trigger, or with no real answer (k = 0), is left as it is, so its most common answer can hold up to the target plus 5 percentage points. A task whose train group has exactly two real answers is instead cut, in every split, so that its two largest answers have equal counts, with no trigger. Rank and label steps: then, in a group where every option value of every row is a number, the rank of the correct option among the sorted values, and after it, in a group where every trained answer is an option label, the label, are each capped by the same cut-and-trigger rule on their own counts (own target, validation included): capped, never evened out, so two labels are cut only when one exceeds 55%, and then only down to 50%. These steps can also cut groups the answer step left whole, including k = 0 groups, and can raise an answer's final share above its target; the run fails if a real answer ends above the target plus 5 percentage points. A train group of at least 20 rows in which one answer holds 90% or more fails the run. PET (exactcellspet) instead cuts each (question type x turn direction) cell to equal counts of its two answers; a PET cell that shows only one answer is removed.

S13 removed 555 rows from this source.

Text-only baselines

Accuracy of guessers that never see an image. For each task the released training rows are split into two fixed halves by a hash of row_uid; each guesser is fitted on one half and scored once on the other (one held-out half, not cross-validation; eval rows below). The reference is chance (the mean of 1 / number of options) where every row is multiple choice, and otherwise the eval-half accuracy of always giving the answer most common in the fit half (when a task's top answers are nearly tied, this need not be the task's most common answer; the line after the table gives that answer's validation score). Accuracies are recounted from the stored rates and eval rows, so they are exact. A task is flagged when a text-only guesser beats its reference by more than 0.15 (for a free-form task, a guesser other than the most common answer). A flagged task can be partly answered from the text alone; an unflagged task passed only these probes, which do not prove the text carries no answer. Report scores on every task next to this baseline.

Guessers: keywords: the most common answer per set of spatial words in the question; last_mentioned: the option named last in the question body; letter_prior: the most common answer letter; majority: the answer most common in the fit half; numeric_offset: where every option is a number, the smallest or largest option plus or minus the offset the correct one sat at most often (last_mentioned on the other rows); option_prior: the option text that won most often when shown; option_rank: where every option is a number, the option at the rank among the sorted values that the correct one held most often (last_mentioned on the other rows); template: the most common answer per question wording (numbers masked, object names kept).

Spot-check (S14)

Pending. The S14 rows are chosen and flagged S14.sampled_qa in meta and preview; the human pass over them has not been signed off yet.

Citation

Please cite the upstream paper and Infinigen (https://github.com/princeton-vl/infinigen), as the VDrop card asks:

@article{yang2026and,
  title={How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning},
  author={Yang, Qian and Sikarwar, Ankur and Le, Huy and Zhang, Le and Shi, Zhuan and Taslakian, Perouz and Agrawal, Aishwarya},
  journal={arXiv preprint arXiv:2605.27310},
  year={2026}
}

@inproceedings{infinigen2024indoors,
  author={Raistrick, Alexander and Mei, Lingjie and Kayan, Karhan and Yan, David and Zuo, Yiming and Han, Beining and Wen, Hongyu and Parakh, Meenal and Alexandropoulos, Stamatis and Lipson, Lahav and Ma, Zeyu and Deng, Jia},
  title={Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month={June},
  year={2024},
  pages={21783-21794}
}

@inproceedings{infinigen2023infinite,
  title={Infinite Photorealistic Worlds Using Procedural Generation},
  author={Raistrick, Alexander and Lipson, Lahav and Ma, Zeyu and Mei, Lingjie and Wang, Mingzhe and Zuo, Yiming and Kayan, Karhan and Wen, Hongyu and Han, Beining and Wang, Yihan and Newell, Alejandro and Law, Hei and Goyal, Ankit and Yang, Kaiyu and Deng, Jia},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={12630--12641},
  year={2023}
}

Provenance

The release files were written by our conversion code (the code repository is not public yet), scripts/convert/export.py at commit 4aaee1f4e946, from build vdrop_r2. The build was made by scripts/convert/run_source.py from the same repository at commit 949af62f8ab7. S10, S12 and S13 ran before the export; reports/export_manifest.json pins every input the export read by SHA-1 (build_manifest_sha1, s10_keep_sha1, s12_assignments_sha1, s13_balanced_keep_sha1).

Every row removed between upstream and this release has one line, with its reason, in reports/: build/dropped.jsonl (rows refused before conversion or dropped by a conversion step); build/quarantine.jsonl (rows set aside by S4c because an automatic check could not match the read-back's conclusion to the label); s10_dropped.jsonl (duplicates removed by S10); s10_label_conflicts.jsonl (rows S10 withheld because another row asks the identical question, options in the same order, of the same images with a different answer); s13_dropped.jsonl (rows removed by answer-prior balancing). known_issues/ lists rows with a measured problem (see Known issues); reports/ also holds the build manifest (absolute paths cut to basenames) and counters, the S14 sample list (s14_sample.tsv: rowuid, task, split) and `exportmanifest.json. Part of [yrlyrl/spatial-mmcot`](https://huggingface.co/datasets/yrlyrl/spatial-mmcot).