luuuulinnnn/scene-extrapolation-benchmark
Scene-Extrapolation Benchmark Suite A unified, honest evaluation of generative novel-view synthesis (NVS) in the regimes where per-scene 3D-Gaussian-Splatting reconstruction fails: Extrapolative — held-out, unobserved views (not interpolation between dense captures). Dynamic — moving foreground content. Long-horizon — chained / loop-closing camera trajectories where drift accumulates. Memory / revisit — recall when the camera returns to a previously-observed pose (do methods… See the full description on the dataset page: https://huggingface.co/datasets/luuuulinnnn/scene-extrapolation-benchmark.
0815
1<section id="coverage">2<style>3#coverage{font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;color:#1a1f29;margin:0 0 8px;}4#coverage h2{font-size:20px;margin:0 0 4px;font-weight:700;letter-spacing:-.01em;}5#coverage .cov-sub{font-size:12.5px;color:#5a6472;margin:0 0 10px;}6#coverage .cov-summary{font-size:12.5px;color:#2c3340;margin:0 0 10px;padding:6px 10px;background:#f4f6f9;border:1px solid #e2e7ee;border-radius:6px;display:inline-block;}7#coverage .cov-summary b{color:#0d1117;}8#coverage .cov-legend{display:flex;flex-wrap:wrap;gap:8px 14px;margin:0 0 12px;font-size:12px;}9#coverage .cov-legend span{display:inline-flex;align-items:center;gap:6px;}10#coverage .chip{display:inline-block;width:13px;height:13px;border-radius:3px;border:1px solid rgba(0,0,0,.12);vertical-align:middle;}11#coverage .chip.s{background:#cdeccf;}12#coverage .chip.r{background:#fde6bd;}13#coverage .chip.n{background:#e3e6ea;}14#coverage .chip.p{background:repeating-linear-gradient(45deg,#fafbfc,#fafbfc 4px,#eef0f3 4px,#eef0f3 8px);}15#coverage .cov-wrap{overflow-x:auto;border:1px solid #dfe4ea;border-radius:8px;}16#coverage table{border-collapse:collapse;width:100%;font-size:11.5px;background:#fff;}17#coverage th,#coverage td{border:1px solid #e6eaef;padding:5px 8px;text-align:center;white-space:nowrap;}18#coverage thead th{background:#eef1f5;font-weight:600;color:#222a35;position:sticky;top:0;}19#coverage th.axis{font-size:12px;letter-spacing:.02em;text-transform:uppercase;color:#3a4452;background:#e4e9f0;}20#coverage th.axis-static{background:#dfeaf6;}21#coverage th.axis-dynamic{background:#e6f3e6;}22#coverage th.axis-long{background:#f3e9da;}23#coverage th.col,#coverage td.method{text-align:left;}24#coverage th.col .tot{display:block;font-weight:400;font-size:10px;color:#6b7480;}25#coverage td.method{background:#f8fafc;font-weight:500;color:#202733;min-width:190px;}26#coverage tr.grouphdr td{background:#dde3ec;text-align:left;font-weight:700;font-size:11.5px;color:#2a3340;letter-spacing:.01em;padding:6px 10px;}27#coverage tr.highlight td.method{background:#fff4cf;font-weight:700;box-shadow:inset 3px 0 0 #e0a800;}28#coverage tr.highlight td{background:#fffdf3;}29#coverage td.c-scored{background:#cdeccf;color:#14512a;font-weight:600;}30#coverage td.c-running{background:#fde6bd;color:#7a4d00;font-weight:600;}31#coverage td.c-na{background:#e3e6ea;color:#5a6472;font-style:italic;}32#coverage td.c-pending{background:repeating-linear-gradient(45deg,#fafbfc,#fafbfc 4px,#eef0f3 4px,#eef0f3 8px);color:#8a929c;font-style:italic;}33#coverage td .reason{display:block;font-size:9.5px;font-weight:400;font-style:italic;color:#7a8492;margin-top:1px;}34#coverage td.c-scored .reason,#coverage td.c-na .reason{color:#69737f;}35</style>36<h2>Coverage matrix — methods × benchmarks</h2>37<p class="cov-sub">Computed honestly from disk (metrics JSONs & generated mp4 outputs under <code>runs/</code>). n = clips scored; gen = mp4 outputs produced.</p>38<p class="cov-summary"><b>20</b> methods × <b>6</b> benchmarks = <b>120</b> cells | ✓ scored: <b>58</b> 🔄 running: <b>5</b> ⏭/— pending+N/A: <b>57</b> (pending 34, N/A 23)</p>39<div class="cov-legend"><span><i class="chip s"></i>✓ scored (n=clips)</span><span><i class="chip r"></i>🔄 running (gen mp4 / total)</span><span><i class="chip n"></i>⏭ N/A (out of protocol)</span><span><i class="chip p"></i>— pending (not run yet)</span></div>40<div class="cov-wrap"><table>41<thead>42<tr><th rowspan="2" class="col">Method</th><th class="axis axis-static" colspan="1">Static</th><th class="axis axis-dynamic" colspan="3">Dynamic</th><th class="axis axis-long" colspan="2">Long-horizon</th></tr>43<tr><th class="col axis-static">DL3DV static (eval50)<span class="tot">total 50</span></th><th class="col axis-dynamic">NVIDIA (secondary)<span class="tot">total 24</span></th><th class="col axis-dynamic">Kubric-4D (GCD)<span class="tot">total 103</span></th><th class="col axis-dynamic">SpatialVID<span class="tot">total 30</span></th><th class="col axis-long">LH-DL3DV drift/loop<span class="tot">total 42</span></th><th class="col axis-long">WorldScore<span class="tot">total 8</span></th></tr>44</thead><tbody>45<tr class="grouphdr"><td colspan="7">Ours (warp+inpaint VACE)</td></tr>46<tr class="methodrow highlight"><td class="method">VACE-14B (ours)</td><td class="c-scored">✓ scored (n=50)</td><td class="c-scored">✓ scored (n=18)</td><td class="c-scored">✓ scored (n=91)</td><td class="c-scored">✓ scored (n=30)</td><td class="c-scored">✓ scored (n=42)</td><td class="c-scored">✓ scored (n=8)</td></tr>47<tr class="methodrow"><td class="method">VACE-14B ft (long-horizon)</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-running">🔄 running (gen: 3/42, ft@4000)</td><td class="c-pending">— pending</td></tr>48<tr class="methodrow"><td class="method">VACE-14B ft (dynamic)</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td></tr>49<tr class="methodrow"><td class="method">VACE-14B ft (mixed)</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td></tr>50<tr class="methodrow"><td class="method">I2V base</td><td class="c-scored">✓ scored (n=34)</td><td class="c-scored">✓ scored (n=21)</td><td class="c-scored">✓ scored (n=90)</td><td class="c-scored">✓ scored (n=29)</td><td class="c-running">🔄 running (gen: 19/42)</td><td class="c-pending">— pending</td></tr>51<tr class="methodrow"><td class="method">I2V ft</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td><td class="c-pending">— pending</td></tr>52<tr class="grouphdr"><td colspan="7">Diffusion baselines</td></tr>53<tr class="methodrow"><td class="method">GEN3C</td><td class="c-scored">✓ scored (n=50)</td><td class="c-scored">✓ scored (n=21)</td><td class="c-scored">✓ scored (n=100)</td><td class="c-scored">✓ scored (n=20)</td><td class="c-scored">✓ scored (n=42)</td><td class="c-scored">✓ scored (n=8)</td></tr>54<tr class="methodrow"><td class="method">Voyager</td><td class="c-scored">✓ scored (n=49)</td><td class="c-scored">✓ scored (n=18)</td><td class="c-scored">✓ scored (n=5)</td><td class="c-scored">✓ scored (n=30)</td><td class="c-scored">✓ scored (n=33)</td><td class="c-scored">✓ scored (n=8)</td></tr>55<tr class="methodrow"><td class="method">FlexWorld</td><td class="c-scored">✓ scored (n=50)</td><td class="c-scored">✓ scored (n=18)</td><td class="c-scored">✓ scored (n=100)</td><td class="c-pending">— pending</td><td class="c-scored">✓ scored (n=42)</td><td class="c-scored">✓ scored (n=8)</td></tr>56<tr class="methodrow"><td class="method">ViewCrafter</td><td class="c-scored">✓ scored (n=36)</td><td class="c-scored">✓ scored (n=21)</td><td class="c-scored">✓ scored (n=100)</td><td class="c-scored">✓ scored (n=29)</td><td class="c-scored">✓ scored (n=42)</td><td class="c-scored">✓ scored (n=8)</td></tr>57<tr class="methodrow"><td class="method">FrameCrafter</td><td class="c-scored">✓ scored (n=50)</td><td class="c-scored">✓ scored (n=21)</td><td class="c-scored">✓ scored (n=100)</td><td class="c-scored">✓ scored (n=27)</td><td class="c-scored">✓ scored (n=41)</td><td class="c-scored">✓ scored (n=8)</td></tr>58<tr class="methodrow"><td class="method">NVS-Solver</td><td class="c-scored">✓ scored (n=45)</td><td class="c-scored">✓ scored (n=18)</td><td class="c-scored">✓ scored (n=100)</td><td class="c-scored">✓ scored (n=30)</td><td class="c-scored">✓ scored (n=42)</td><td class="c-scored">✓ scored (n=8)</td></tr>59<tr class="methodrow"><td class="method">ViewExtrapolator</td><td class="c-scored">✓ scored (n=50)</td><td class="c-scored">✓ scored (n=18)</td><td class="c-pending">— pending</td><td class="c-scored">✓ scored (n=30)</td><td class="c-scored">✓ scored (n=32)</td><td class="c-scored">✓ scored (n=8)</td></tr>60<tr class="methodrow"><td class="method">TrajCrafter</td><td class="c-scored">✓ scored (n=48)</td><td class="c-scored">✓ scored (n=14)</td><td class="c-scored">✓ scored (n=67)</td><td class="c-scored">✓ scored (n=24)</td><td class="c-scored">✓ scored (n=42)</td><td class="c-scored">✓ scored (n=8)</td></tr>61<tr class="grouphdr"><td colspan="7">Per-scene 3DGS</td></tr>62<tr class="methodrow"><td class="method">Shape-of-Motion (4D-GS)</td><td class="c-na">⏭ N/A<span class="reason">per-scene 4D-GS, not applicable</span></td><td class="c-scored">✓ scored (n=21)</td><td class="c-scored">✓ scored (n=15)</td><td class="c-running">🔄 running (gen: 30/30)</td><td class="c-running">🔄 running (gen: 42/42)</td><td class="c-na">⏭ N/A<span class="reason">per-scene 4D-GS, not applicable</span></td></tr>63<tr class="grouphdr"><td colspan="7">Long-horizon SOTA (world models)</td></tr>64<tr class="methodrow"><td class="method">Lyra-2</td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-running">🔄 running (gen: 24/42)</td><td class="c-pending">— pending</td></tr>65<tr class="methodrow"><td class="method">Matrix3D</td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-pending">— pending (queued)<span class="reason">queued</span></td><td class="c-pending">— pending</td></tr>66<tr class="methodrow"><td class="method">SpatialMem</td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-pending">— pending (spmem running, no outputs yet)<span class="reason">spmem running, no outputs yet</span></td><td class="c-pending">— pending</td></tr>67<tr class="methodrow"><td class="method">VMem</td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-pending">— pending (gated)<span class="reason">gated</span></td><td class="c-pending">— pending</td></tr>68<tr class="methodrow"><td class="method">MatrixGame-3</td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">LH world-model, eval on LH only</span></td><td class="c-na">⏭ N/A<span class="reason">run broken</span></td><td class="c-pending">— pending</td></tr>69</tbody></table></div>70</section>