NagaYu/saccade-egomotion-bench

Saccade ego-motion benchmark

The stream, the raw decision signals, and the per-frame measurements behind **Saccade** — an always-on edge VLM that re-encodes only the image patches whose change ego-motion cannot explain.

This dataset exists so the central claim can be checked without running our code.

• —💻 Code: https://github.com/NagaYu/saccade
• —🤖 Model: https://huggingface.co/NagaYu/saccade-predictor
• —🚀 Demo: https://huggingface.co/spaces/NagaYu/saccade

The claim, in one number

On the segments with predictable camera motion (walk + turn), at an identical threshold τ = 0.06:

Same frames, same threshold — only the reference frame differs. A moving camera changes almost every pixel, so frame-differencing cannot thin the workload during steady motion; subtract the ego-motion first and a static world's residual collapses to nearly nothing.

Verify it yourself in four lines:

from datasets import load_dataset
import numpy as np

sig = load_dataset("NagaYu/saccade-egomotion-bench", "signals", split="train").to_pandas()
walk = sig[sig.segment.isin(["walk", "turn"])]
print((np.stack(walk.residual_same_position)        > 0.06).mean())  # ~0.474
print((np.stack(walk.residual_motion_compensated)   > 0.06).mean())  # ~0.003

Configs

frames — the stream itself (144 rows)

The scripted ego-motion video, so another method can be benchmarked on the exact same input.

signals — the two competing decision signals (143 rows)

The direct evidence. Per frame, the per-patch residual map under both rules.

Residuals are mean absolute pixel differences in [0,1] on a 14×14 patch grid (row-major; reshape(14, 14) to get a map).

measurements — per-frame energy and quality (432 rows = 144 × 3 conditions)

Headline (steady-state, excluding 6 warm-up frames, budget 20 mW):

How the stream was made

Deterministically generated (saccade.stream.synthetic_walking_stream, seed 0): a window panning over a large static textured canvas, scripted into four regimes —

• —static — camera still, world still (both methods should skip; fair region)
• —walk — smooth translation across a static world (predictable ego-motion)
• —turn — faster translation plus mild rotation (harder ego-motion)
• —event — camera still, an independently moving object crosses (genuine change; every method must spend here)

Limitations — please read

• —This is synthetic video, not natural footage. It is a controlled apparatus for the steady-motion regime, chosen so the result is exactly reproducible. It is deliberately not evidence about natural scenes with parallax, motion blur, rolling shutter, exposure changes or non-rigid motion. Treat conclusions as being about the mechanism, not about wearable cameras in the wild.
• —fidelity is patch-token cosine similarity against a Full encoder — a proxy for downstream task quality, not a task metric like VQA accuracy.
• —Energy is an analytic FLOP model (FLOPs × 1 pJ/FLOP), not a power-meter measurement. Relative comparisons and the budget guarantee are sound; absolute Joules are model-dependent.
• —measurements uses the SyntheticBackbone (a frozen random ViT-S/16-class patch encoder), chosen because each patch embedding is genuinely independent so subset encoding realizes real savings. Numbers on a real ViT differ; see the repo for DINOv2-small results.

Reproduce

git clone https://github.com/NagaYu/saccade && cd saccade
pip install -r requirements.txt
python hf/export_dataset.py     # rebuilds every config from scratch

License

MIT