resoajoe/sidecar-demo

Per-frame measurement records carried inside an H.264 stream as SEI NAL units (userdataunregistered, payloadType 5, a 16-byte UUID and one Sidecar/1 JSON envelope per access unit). Tab A fetches a 20-second 640x360 clip, demuxes the mp4 in plain JavaScript (its own box parser: stsz, stsc, stco, stts, ctts), extracts the envelope from every sample, recovers display order from the container's composition offsets (B-frames reorder), and draws the measurements over the picture in sync with playback. Each envelope names what produced every row (by.model, by.kind) and carries the answerability gate whose passed is the only field a consumer may read as a validity signal, plus a 64-bit fingerprint of the frame. Tab B is the stronger claim: it loads sidecar_transcoded.mp4, the same clip re-encoded at 480x270 with libx264 CRF 28, which carries 0/600 SEI payloads because a re-encode destroys them; the page decodes all 600 frames in the browser (WebCodecs VideoDecoder, hashing the Y plane straight from the decoder, falling back to seek-and-draw on a canvas where WebCodecs is missing), fingerprints each one, loads the 600 envelopes from sidecar_rows.json as a receiver might hold them from a data channel, and re-binds them to frames by banded Needleman-Wunsch alignment of the two fingerprint sequences (window 60, gap 6). Measured in Chromium on this machine: 600/600 rows re-bound, every row to the frame it came from, median Hamming 0, decode and hash 1.8 s, alignment 18 ms.

Measured on this file with ffmpeg 8.0 / x264 on one machine: every copy path keeps every payload (remux mp4 -> mkv -> mp4, MPEG-TS remux, HLS with 2 s TS segments, fragmented mp4: 600/600) and every re-encode drops all of them (0/600). Fingerprint survival on the same material: the same frame before and after the 480x270 CRF 28 re-encode differs by 0 bits (median, max 4), consecutive frames of this moving scene by 4 bits; alignment rebinds 600/600 where nearest-neighbour matching gets 592/600, because the survival tolerance of the hash is the same size as the distance between neighbouring frames. The named failure is on the page: on a static scene the fingerprint carries no ordering information and the alignment is arbitrary within a run, which the ambiguous flag detects when read together with a consecutive-frame Hamming median of 0 - and the page refuses with "unknown / refused: static scene" if it sees that combination. Hardware encoders, RTSP/RTP pipelines, WebRTC SFUs, HLS through a CDN and HEVC were not tested; the clip is a rendered synthetic scene, not a recording, and its three measurement rows are cheap scalars (mean luminance, Laplacian variance, changed 16x16 blocks), not nano models.

Everything runs on the visitor's device: the page loads only its own files, no CDN and no server. The footer prints parity against the Python reference (nanolab.sidecar1): the browser's fingerprint self-test against the spec's check vector aa0a7f817e817e85, 10/10 SEI payloads byte-exact against nanolab.sidecar.extract, and, for the first 20 decoded frames, max Hamming 0/64 between the JavaScript and Python fingerprints with 20/20 rebind agreement. Spec and reference implementation: SIDECAR1.md, sidecar1.py, sidecar.py; the browser side is sidecar_demux.js and sidecar_fp.js, both also runnable in Node (node parity_check.js).

Clips

• —Spec (synthetic) — the original moving test pattern used for byte-exact parity and survival tables.
• —Real drive — 12 s author-filmed Tesla cabin crop around an overpass (10 fps fill-crop to 640×360). Dashboard-forward; windshield mostly occluded. Sidecar/1 rows include scalar dashboard measurements plus a refused road.lane gate under Johnson recognise (8 px). No faces. Built with build_real.py in the Resoa workspace.