luuuulinnnn/scene-extrapolation-benchmark

Scene-Extrapolation Benchmark Suite

A unified, honest evaluation of generative novel-view synthesis (NVS) in the regimes where per-scene 3D-Gaussian-Splatting reconstruction fails:

• —Extrapolative — held-out, unobserved views (not interpolation between dense captures).
• —Dynamic — moving foreground content.
• —Long-horizon — chained / loop-closing camera trajectories where drift accumulates.
• —Memory / revisit — recall when the camera returns to a previously-observed pose (do methods remember the scene, or re-hallucinate it?).

One-line framing: diffusion in the regimes 3DGS fails — extrapolative / dynamic / long-horizon / memory.

This dataset repo is a self-contained, offline-playable visualization of the suite as a live snapshot. Open viz_pkg/index.html by double-click (or browse index.html in the repo) — all paths are relative, videos are embedded H.264, no server needed. A zipped copy is provided as scene_extrapolation_benchmark_viz.zip.

What's inside

viz_pkg/
  index.html                 self-contained page (header + 4 sections, inlined CSS/JS/SVG)
  README.md                  this file
  sections/                  the source HTML fragments + the coverage snapshot JSON
    difficulty.html          the headline Kubric-4D difficulty-curve charts (inline SVG)
    coverage.html / coverage_data.json
    quantitative.html
    qualitative.html
    infer_time.json          per-(method,benchmark) avg inference wall-clock (from SLURM sacct)
  assets/qual/<bench>/<cid>/*.mp4   FULL-RESOLUTION individual per-method clips (H.264 crf 20,
                                    NATIVE resolution — GEN3C 1280×704, ViewCrafter 1024×576,
                                    NVS/SoM 960×540, FrameCrafter/Ours 832×480, Kubric 576×384;
                                    NO downscaling). Laid out in the page as a CSS grid of
                                    lazy-loaded <video> elements (≤4 per row), replacing the old
                                    downscaled composited-grid panels.
  assets/qual/{mv,mb}_*/<cid>/       REVISIT / MEMORY qualitative panels (6 benchmarks:
    *.mp4                            multi_visit + moving_back × dl3dv / spatialvid / kubric4d,
    return/*.png                     ~10 clips each). Per clip: a full-res per-method video grid
                                     surfacing the memory methods (Shape-of-Motion, GEN3C, SpMem,
                                     Voyager, Lyra-2, MatrixGame-3) + Ours + honest revisit-GT,
                                     PLUS a return-frame comparison (return/<method>_pose*.png =
                                     the model's generated frame at the return pose, next to
                                     obsgt_pose*.png = the ORIGINAL observed GT there) with the
                                     reused return/self PSNR·SSIM·LPIPS numbers.

Synchronized playback. Every qualitative panel (static, dynamic, long-horizon and the revisit panels) has a per-panel sync controller: one Play all toggle + a shared scrubber that starts / pauses / seeks all tiles in lockstep and loops them together, so the methods are always compared at the same timestamp while each tile keeps its own native resolution + download.

The page has four parts:

1. 1.Kubric-4D difficulty curve (headline) — the primary, confound-free dynamic story. Inline-SVG line charts of metric-vs-camera-angle-bin (x = 30/60/90/120/180°, one line per method) for full-frame PSNR, dynamic-masked mPSNR, LPIPS, and a camera-trajectory error (COLMAP ATE). Every method degrades as the viewpoint widens; the gap at the hard 120–180° back-views is the real test. Ours (vace14b) is the bold green line. Charts are pure inline SVG — no CDN, fully offline.
2. 2.Coverage matrix — what we ran. Rows = all methods (grouped: Ours · diffusion baselines · per-scene 3D-GS · long-horizon world models), columns = the 6 benchmarks across the 3 axes. Each cell is ✓ scored (n=N) / 🔄 running / ⏭ N/A (reason) / — pending.
3. 3.Quantitative — per-benchmark metric tables read directly from runs/<bench>_metrics/<method>.json. Static: PSNR/SSIM/LPIPS (hole/extrap + full-frame), FID, TSED, MEt3R, COLMAP. Dynamic: Kubric-4D clean-GT (primary, angle-binned 30/60/90/120/180°, 4 metric families) + NVIDIA dynamic-region-masked mPSNR/mSSIM/mLPIPS (secondary) + SpatialVID. Long-horizon: drift (ATE / rotation / loop-closure) + the WorldScore aggregate — now with the LH-SOTA rows (lyra2, matrixgame3) scored, the vace14b_ft_lh ft@4000 row filled, and a §4c base-vs-finetune head-to-head. A dedicated Memory / revisit section adds the mv_* (revisit-consistency) and mb_* (return-fidelity) tables across Kubric-4D / DL3DV / SpatialVID. Best per column is bold; the Ours (vace14b) row is highlighted. Each per-benchmark table now also carries an `infer s/clip` column (last column) and there is a dedicated §5 Average inference time summary (per-method s/clip with s/frame, sorted fastest→slowest). These are honest wall-clock numbers from SLURM `sacct` `ElapsedRaw` summed over the COMPLETED generation array tasks / clips produced (NOT mtime deltas, which parallel array writes make meaningless). Caveats rendered inline: it is wall-clock incl. one-time model-load amortization; clip lengths differ per benchmark so s/frame is the cross-benchmark-fair number; per-scene shape_of_motion includes the full 4D-GS optimization (flagged per-scene); methods with no sacct record show — (never fabricated). The spread runs from fast feed-forward (Lyra-2 / Matrix-Game ~1.5–2 s/frame) through autoregressive-memory (Voyager 13–100 s/frame) to per-scene Shape-of-Motion (~62 s/frame).
4. 4.Qualitative — per representative clip, a CSS grid of individual full-resolution videos (≤4 per row): observed input | warp / point-cloud | key baselines (incl. FrameCrafter) | Ours | GT, including long-horizon closed-loop / revisit clips. Each <video> plays its method's output at native resolution (no downscaling; object-fit: contain normalizes the grid box) and is lazy-loaded (preload="none" + IntersectionObserver load-on-scroll) inside per-benchmark collapsible sections, so the page stays responsive despite hundreds of clips. Clip sets: eval50 (12), NVIDIA (12), SpatialVID (12), Kubric-4D (25, grouped by 30/60/90/120/180° bin), LH-DL3DV (12, grouped by trajectory family). Honest GT handling preserved: synthetic-orbit GT dropped, closed-loop GT relabeled "observed-only", unscored methods tagged (metrics pending).

Benchmarks (12 across 4 axes)

Memory axis (NEW): both families score only on real GT (revisit/return legs carry true GT; synthetic away-legs are masked upstream). Headline — explicit geometric caching, not learned memory, enables return-fidelity — now confirmed across all three scene types: static-real DL3DV, synthetic Kubric-4D, AND dynamic-real SpatialVID (9 methods scored per family, 52 cells; spmem and the warp-conditioned framecrafter now folded in on all six memory benchmarks). Per-scene 4D-GS (shape_of_motion) stores the scene and degrades only −0.8 dB (DL3DV) / −1.0 dB (SpatialVID); the best forward generator is gen3c with an explicit 3D point cache, the only generator that stays low on every scene type (−17.0 dB DL3DV / −13.8 dB Kubric / −17.5 dB SpatialVID). The dedicated learned spatial-memory model spmem collapses −30.0 / −24.1 / −32.0 dB (DL3DV / Kubric / SpatialVID) — among the worst on all three, uniform across static, synthetic and dynamic, so it is not a domain gap. The memoryless single-shot generators (lyra2 / VACE / viewcrafter / nvs_solver) all collapse ≈ −20 to −32 dB, and spmem sits inside that band. (Matrix-Game 3.0's tiny degrade is an artifact of uniformly-low quality, return-PSNR ≈ 11 — not memory.) Today's strongest extrapolators have no working learned memory; explicit geometry carries it.

DyCheck (iPhone handheld) was dropped: its portrait aspect ratio doesn't fit the landscape pipeline, so it would be an unfair comparison. The confound-free Kubric-4D clean-GT anchor (with perfect ground-truth, no warp confound) is the primary dynamic story; NVIDIA is the secondary check.

Methods

• —Ours (warp+inpaint VACE): vace14b (Wan2.1-VACE-14B, depth-warp control → inpaint) and its finetunes vace14b_ft_lh / vace14b_ft_dyn / vace14b_ft_mixed, plus i2v_base / i2v_ft.
• —Diffusion baselines: gen3c, voyager, flexworld, viewcrafter, framecrafter, nvs_solver, viewextrapolator, trajcrafter.
• —Per-scene 3D-GS: shape_of_motion (monocular 4D Gaussian fit).
• —Long-horizon world models: lyra2, matrix3d, spmem, vmem, matrixgame3.

Honesty / snapshot caveats

This is a live snapshot (2026-06-29) — several jobs are still running. Cells with no scored <method>.json render as running / pending / N/A, never as a fabricated number. Notable state at snapshot time: the long-horizon SOTA rows lyra2 and matrixgame3 are now scored on the LH drift probe, and vace14b_ft_lh is scored at `ft@4000` (see the §4c base-vs-finetune head-to-head); spmem still has no LH metrics json (long-horizon roll still generating, left running). The NEW Memory / revisit axis now carries 9 methods scored per benchmark (52 cells total) across all six families, with spmem and framecrafter both folded in on all three scene types; vmem/FlexWorld/TrajCrafter/ViewExtrapolator/Voyager and spmem on the still-generating long-horizon set are still rolling out there. The vace14b recon metrics on NVIDIA / Kubric summary are still shown as running (warp pipeline still scoring) rather than as numbers.

Reproduction

Metrics are aggregated from per-clip rows by code/build_bench.py into viz/bench_data.json; the Kubric-4D per-angle-bin difficulty curve (inline SVG) is built by code/build_kubric_curve.py from the by_bin aggregates; the section fragments and the video panels are built on top, then assembled into index.html. The inference-time column/summary is computed by code/compute_infer_time.py (SLURM sacct → sections/infer_time.json) and injected into the quantitative tables by code/add_infer_column.py (additive, re-runnable, idempotent).