AntonioJun/workspace

Spatial Code VSI-Bench Workspace

This workspace evaluates VSI-Bench question answering with several input regimes: raw video frames, perceived spatial codes from SAM3 + Depth Anything 3 caches, ground-truth spatial codes from dataset annotations, and a deterministic symbolic solver. The code is organized so important outputs are reproducible from fixed inputs, fixed packages, fixed model checkpoints, and fixed SAM3/DA3 caches.

The repository intentionally separates three artifact classes:

• —source code and tests in /workspace
• —data, caches, and model checkpoints under /root/data and /root/models
• —generated results and reports under /root/results and /workspace/reports

Quick Start

# Install packages, clone external repos, download data/models where allowed.
./setup.sh

# Non-interactive setup with a Hugging Face token.
HF_TOKEN=hf_xxx ./setup.sh -y

# Run all unit tests. Tests do not require data, results, or checkpoints.
python -m pytest -q

# Check style and syntax.
python -m black --check /workspace
python -m compileall -q /workspace

Useful setup variants:

./setup.sh --skip-models       # packages + repos + data, but no model checkpoints
./setup.sh --skip-data         # packages + model repos/checkpoints, but no VSI-Bench data
./setup.sh --skip-workspace    # do not sync workspace source from the backup dataset
./setup.sh --with-caches       # also sync cached SAM3/DA3 artifacts from the backup dataset
./setup.sh --with-segvggt      # clone/install optional SegVGGT support
./setup.sh --force             # recreate/re-download targets

After setup, use:

source /root/.venv/bin/activate
source /root/vsi-env.sh

/root/vsi-env.sh sets the path variables the code expects, including VSI_CODES="/root/data/spatial codes".

Runtime Paths

The default paths can be overridden by environment variables, but these are the expected locations:

Generated/cache folders in /workspace such as __pycache__, .pytest_cache, .cache, .ipynb_checkpoints, reports, and results are not source modules. They are not required by tests.

Reproducibility Contract

Assuming the same inputs, packages, checkpoints, command/config, and frozen SAM3/DA3 caches:

• —spatial-code generation from fixed SAM3/DA3 caches is code-level reproducible
• —VLM calls use fixed prompts and deterministic decoding config
• —Harness outputs are organized by every meaningful config axis to avoid collisions
• —reports have deterministic manifest timestamps by default
• —tests use synthetic fixtures and do not require data, results, caches, or checkpoints

Regenerating SAM3/DA3 raw caches themselves is not guaranteed deterministic.

Workflow Overview

1. 1.Run perception caches if needed with inference.run or inference.launch.
2. 2.Build perceived spatial codes with encoder.run or encoder.launch.
3. 3.Run VLM harnesses A-C or symbolic harness F.
4. 4.Regenerate reports with analysis.letters_reports.
5. 5.Back up selected outputs with backup.py.

Command Reference

Every CLI supports --help; that is the authoritative flag list. The examples below show the intended interfaces and common combinations.

Setup And Backup

./setup.sh [--skip-models] [--skip-data] [--skip-workspace] [--with-caches] [--with-segvggt] [--force]

python backup.py all --repo owner/dataset --dry-run
python backup.py code --repo owner/dataset
python backup.py reports --repo owner/dataset
python backup.py spatial-codes --repo owner/dataset
python backup.py A,B,C --repo owner/dataset

backup.py targets are defined in TARGETS: code, reports, spatial-codes, Dry-run mode does not import or call huggingface_hub.

Inference: Raw Model Caches

# One scene, one model backend.
python -m inference.run SCENE --model sam3 --tracking tracking --input uniform --frames 32 --device cuda
python -m inference.run SCENE --model depth-anything-3 --depth metric --input uniform --frames 32 --device cuda

# Batch across selected or all manifest scenes.
python -m inference.launch --model sam3 --tracking tracking --input uniform --frames 32 --scenes scene1,scene2
python -m inference.launch --model depth-anything-3 --depth metric --input uniform --frames 32

Important files:

• —inference/__init__.py: paths, cache path helpers, frame-selection vocabulary
• —inference/adapters.py: SAM3, DA3, DA3 metric, optional SegVGGT adapters
• —inference/prompts.py: dataset-specific object prompts for SAM3
• —inference/run.py: single-scene cache writer
• —inference/launch.py: multi-scene worker launcher

Encoder: Spatial Codes

# One scene from existing SAM3/DA3 caches.
python -m encoder.run SCENE --depth metric --tracking tracking --input uniform --frames 32

# Batch over every manifest scene with required caches.
python -m encoder.launch --depth metric --tracking tracking --input uniform --frames 32

# Full-video mode; depth and tracking remain independently selected.
python -m encoder.run SCENE --depth relative --tracking tracking --video
python -m encoder.launch --depth relative --tracking tracking --video

Important files:

• —encoder/config.py: path construction and validated axes
• —encoder/adapters.py: raw-cache readers into canonical geometry
• —encoder/geometric.py: spatial-code construction math
• —encoder/render.py: writes final explicit spatial-code JSON
• —encoder/run.py: loads/verifies raw caches and builds one scene
• —encoder/launch.py: CPU-parallel batch driver

Experiments: Geometry Hypotheses

# List available hypothesis forks.
python -m experiments.run --list

# Build one hypothesis spatial code from existing caches only.
python -m experiments.run SCENE --hypothesis "Compute Gravity Before Building Object Instances" --depth metric --tracking tracking --input uniform --frames 64 --format explicit

# Batch all scenes with existing combined or native SAM3/DA3 caches.
python -m experiments.launch --hypothesis "Compute Gravity Before Building Object Instances" --depth metric --tracking tracking --input uniform --frames 64 --format explicit

# Evaluate generated experiment spatial codes with the symbolic scorer.
python -m experiments.evaluate --hypothesis "Compute Gravity Before Building Object Instances" --depth metric --tracking tracking --input uniform --frames 64 --format explicit --quiet --errors

Files:

• —experiments/__init__.py: experiments package marker
• —experiments/README.md: experiment workflow notes
• —experiments/EXPERIMENT FINDINGS.md: single consolidated findings report
• —experiments/hypotheses.md: hypothesis index and notes
• —experiments/config.py: experiment-local path construction
• —experiments/adapters.py: build-call adapter for explicit and compact hypothesis forks
• —experiments/loader.py: dynamic loader for human-readable hypothesis filenames
• —experiments/run.py: one-scene cache-only hypothesis spatial-code builder
• —experiments/launch.py: batch launcher over scenes with existing caches
• —experiments/evaluate.py: symbolic evaluation of experiment spatial codes
• —experiments/hypotheses/*.py: standalone geometry hypothesis forks; each exposes build_spatial_code() and dump_spatial_code()

Symbolic Solver

# One scene.
python -m symbolic.run SCENE --depth metric --tracking tracking --input uniform --frames 32 --format explicit

# Batch over available spatial codes.
python -m symbolic.launch --depth metric --tracking tracking --input uniform --frames 32 --format explicit

# Interactive/debug solver entry point.
python -m symbolic.solver

Important files:

• —symbolic/adapters.py: adapts compact/explicit codes into solver shape
• —symbolic/solver.py: deterministic VSI-Bench answering logic
• —symbolic/run.py: scores one scene and writes per-question JSON
• —symbolic/launch.py: multi-scene symbolic orchestration

Harness A: Frames or Native Video

The shared policy is configured in harness/A/__init__.py as QUESTION_PROTOCOLS. Edit that mapping once to switch either question group. Both groups write to the same configuration folder; each result JSON records its own question_group and protocol.

python -m harness.A.run --model qwen3.5-4b --frame-selection uniform --frames 32 --scene SCENE
python -m harness.A.launch --model qwen3.5-4b --frame-selection uniform --frames 32 --scenes scene1,scene2
python -m harness.A.sweep --models all --frame-selections all --frames 16,32
python -m harness.A.run --model qwen3.5-4b --video --scene SCENE
python -m harness.A.launch --model qwen3.5-4b --video
python -m harness.A.sweep --models all --video

Question protocols are hardcoded centrally: numerical questions use base; multiple-choice questions use thinking. --reasoning-budget and --force-budget affect only thinking questions. Other flags include --results-dir, --rebuild, --limit, --device, and --no-write.

Files:

• —harness/A/__init__.py: model paths, protocol constants, result root
• —harness/A/frames.py: uniform/selective frame sampling
• —harness/A/models.py: VLM adapters and deterministic generation config
• —harness/A/prompts.py: VSI-Bench prompt text for frame inputs
• —harness/A/run.py: one model/config/scene
• —harness/A/launch.py: persistent GPU workers for one config
• —harness/A/sweep.py: grid over models, frame selections, frame counts

Harness B: Spatial Code Text Only

python -m harness.B.run --model qwen3.5-4b --depth metric --tracking tracking --input-selection uniform --frames 32 --scene SCENE
python -m harness.B.launch --model qwen3.5-4b --depth metric --tracking tracking --input-selection uniform --frames 32
python -m harness.B.sweep --models all --depths metric --trackings tracking --input-selections uniform --frames 32
python -m harness.B.run --model qwen3.5-4b --depth metric --tracking tracking --video --scene SCENE
python -m harness.B.launch --model qwen3.5-4b --depth metric --tracking tracking --video
python -m harness.B.sweep --models all --depths metric --trackings tracking --video

Question protocols use the same hardcoded numerical=base, multiple-choice=thinking policy. Budget flags affect only thinking questions. Other common flags are --results-dir and --rebuild.

Files:

• —harness/B/__init__.py: B constants and result root
• —harness/B/spatial_codes.py: loads perceived explicit JSON
• —harness/B/prompts.py: spatial-code-only prompt construction
• —harness/B/run.py: one model/config/scene
• —harness/B/launch.py: persistent GPU workers for one config
• —harness/B/sweep.py: grid over model/depth/tracking/input/frame axes

Harness C: Frames Plus Spatial Code

python -m harness.C.run --model qwen3.5-4b --depth metric --tracking tracking --input-selection uniform --frames 32 --spatial-code-source frames --spatial-code-input-selection selective --spatial-code-frames 64 --scene SCENE
python -m harness.C.launch --model qwen3.5-4b --depth metric --tracking tracking --input-selection uniform --frames 32 --spatial-code-source frames --spatial-code-input-selection selective --spatial-code-frames 64
python -m harness.C.sweep --models all --depths metric --trackings tracking --input-selections uniform --frames 32 --spatial-code-sources frames --spatial-code-input-selections selective --spatial-code-frames 64
python -m harness.C.run --model qwen3.5-4b --depth metric --tracking tracking --input-selection uniform --frames 32 --spatial-code-source video --scene SCENE
python -m harness.C.launch --model qwen3.5-4b --depth metric --tracking tracking --input-selection uniform --frames 32 --spatial-code-source video
python -m harness.C.sweep --models all --depths metric --trackings tracking --input-selections uniform --frames 32 --spatial-code-sources video

Files:

• —harness/C/__init__.py: C constants and result root
• —harness/C/prompts.py: combined frames + code prompt construction
• —harness/C/run.py: one model/config/scene
• —harness/C/launch.py: persistent GPU workers for one config
• —harness/C/sweep.py: grid over independent visual-input and spatial-code-input axes

Harness F: Symbolic Solver As A Harness

python -m harness.F.run --source perceived --depth metric --tracking tracking --input-selection uniform --frames 32 --scene SCENE
python -m harness.F.launch --source perceived --depth metric --tracking tracking --input-selection uniform --frames 32
python -m harness.F.sweep --sources perceived --depths metric --trackings tracking --input-selections uniform --frames 32
python -m harness.F.run --source perceived --depth metric --tracking tracking --video --scene SCENE
python -m harness.F.launch --source perceived --depth metric --tracking tracking --video
python -m harness.F.sweep --sources perceived --depths metric --trackings tracking --video

Files:

• —harness/F/__init__.py: F constants and result root
• —harness/F/run.py: symbolic scoring path as a harness
• —harness/F/launch.py: thin launch entry point to run.main
• —harness/F/sweep.py: grid over symbolic source/config axes

Analysis And Reports

Current analysis is centered on report generation through analysis.letters_reports and letter-specific wrappers.

# Regenerate the standard report set from existing results.
python -m analysis.letters_reports \
  --cell A=/root/results/A \
  --cell B=/root/results/B \
  --cell C=/root/results/C \
  --cell D=/root/results/D \
  --cell F=/root/results/F \
  --output-dir /workspace/reports \
  --spatial-codes-dir "/root/data/spatial codes"

# Generate one letter report.
python -m analysis.A_reports --results-dir /root/results/A --output-dir /workspace/reports
python -m analysis.B_reports --results-dir /root/results/B --output-dir /workspace/reports
python -m analysis.C_reports --results-dir /root/results/C --output-dir /workspace/reports
python -m analysis.F_reports --results-dir /root/results/F --output-dir /workspace/reports

Files:

• —analysis/letters_reports.py: shared record discovery, matched summaries, report export
• —analysis/A_reports.py: wrapper for A report
• —analysis/B_reports.py: wrapper for B report
• —analysis/C_reports.py: wrapper for C report
• —analysis/F_reports.py: wrapper for F report

Result Layouts

Full Source File Index

Top Level

analysis/

encoder/

harness/

inference/

symbolic/

Test File Index

The tests mirror source folders. They are written to run without real data, results, or model checkpoints.

tests/

• —tests/__init__.py
• —tests/conftest.py

tests/test_A/

• —tests/test_A/__init__.py
• —tests/test_A/conftest.py
• —tests/test_A/test_A.py
• —tests/test_A/test_frames.py
• —tests/test_A/test_launch.py
• —tests/test_A/test_models.py
• —tests/test_A/test_prompts.py
• —tests/test_A/test_run.py
• —tests/test_A/test_sweep.py

tests/test_B/

• —tests/test_B/__init__.py
• —tests/test_B/conftest.py
• —tests/test_B/test_B.py
• —tests/test_B/test_launch.py
• —tests/test_B/test_prompts.py
• —tests/test_B/test_run.py
• —tests/test_B/test_spatial_codes.py
• —tests/test_B/test_sweep.py

tests/test_C/

• —tests/test_C/__init__.py
• —tests/test_C/conftest.py
• —tests/test_C/test_C.py
• —tests/test_C/test_launch.py
• —tests/test_C/test_prompts.py
• —tests/test_C/test_run.py
• —tests/test_C/test_sweep.py

tests/test_F/

• —tests/test_F/__init__.py
• —tests/test_F/conftest.py
• —tests/test_F/test_F.py
• —tests/test_F/test_launch.py
• —tests/test_F/test_run.py
• —tests/test_F/test_sweep.py

tests/test_analysis/

• —tests/test_analysis/__init__.py
• —tests/test_analysis/conftest.py
• —tests/test_analysis/test_A_reports.py
• —tests/test_analysis/test_B_reports.py
• —tests/test_analysis/test_C_reports.py
• —tests/test_analysis/test_F_reports.py
• —tests/test_analysis/test_analysis.py
• —tests/test_analysis/test_letters_reports.py

tests/

• —tests/test_backup.py

tests/test_encoder/

• —tests/test_encoder/conftest.py
• —tests/test_encoder/test_adapters.py
• —tests/test_encoder/test_config.py
• —tests/test_encoder/test_encoder.py
• —tests/test_encoder/test_geometric.py
• —tests/test_encoder/test_ground_truth.py
• —tests/test_encoder/test_init.py
• —tests/test_encoder/test_launch.py
• —tests/test_encoder/test_render.py
• —tests/test_encoder/test_run.py

tests/test_experiments/

• —tests/test_experiments/__init__.py
• —tests/test_experiments/conftest.py
• —tests/test_experiments/test_config.py
• —tests/test_experiments/test_evaluate.py
• —tests/test_experiments/test_experiments.py
• —tests/test_experiments/test_hypotheses.py
• —tests/test_experiments/test_launch.py
• —tests/test_experiments/test_loader.py
• —tests/test_experiments/test_run.py

tests/test_harness/

• —tests/test_harness/__init__.py
• —tests/test_harness/conftest.py
• —tests/test_harness/test_harness.py

tests/test_inference/

• —tests/test_inference/conftest.py
• —tests/test_inference/test_adapters.py
• —tests/test_inference/test_inference.py
• —tests/test_inference/test_init.py
• —tests/test_inference/test_launch.py
• —tests/test_inference/test_prompts.py
• —tests/test_inference/test_run.py

tests/test_symbolic/

• —tests/test_symbolic/conftest.py
• —tests/test_symbolic/test_adapters.py
• —tests/test_symbolic/test_launch.py
• —tests/test_symbolic/test_run.py
• —tests/test_symbolic/test_solver.py
• —tests/test_symbolic/test_symbolic.py

Maintenance Rules

When adding a Python source file, add the corresponding test in the mirrored test folder. For example:

Every tests/test_<folder>/ directory should contain both conftest.py and test_<folder>.py. Tests should use temporary files, monkeypatching, and synthetic fixtures rather than relying on /root/data, /root/results, /workspace/data, or model checkpoints.

Before handing off changes, run:

python -m black /workspace
python -m black --check /workspace
python -m compileall -q /workspace
python -m pytest -q