uv-scripts/video

1---2viewer: false3tags:4  - uv-script5  - video6  - video-text-to-text7  - video-captioning8  - temporal-grounding9---10 11# Video12 13Scripts for captioning and temporally grounding video files using HF Buckets and Jobs.14 15What the output looks like — a frame from *Joan Avoids a Cold* (1947, Prelinger Archives) with the event Marlin-2B produced for that moment:16 17![Example: film frame with its timestamped event caption](example-card.jpg)18 19## Quick Start20 21Scripts run directly from their Hub URL — no clone or local checkout needed:22 23```bash24# Caption every video in a bucket: dense scene captions + timestamped events25hf jobs uv run --image vllm/vllm-openai:latest --flavor a10g-small \26    -s HF_TOKEN \27    -v hf://buckets/user/my-videos:/input:ro \28    https://huggingface.co/datasets/uv-scripts/video/raw/main/marlin-caption.py \29    /input hf://buckets/user/my-videos/captions30 31# Temporal grounding: when does an event happen?32hf jobs uv run --image vllm/vllm-openai:latest --flavor a10g-small \33    -s HF_TOKEN \34    -v hf://buckets/user/my-videos:/input:ro \35    https://huggingface.co/datasets/uv-scripts/video/raw/main/marlin-caption.py \36    /input hf://buckets/user/out --find "a person enters the room"37```38 39## Scripts40 41### marlin-caption.py42 43Runs [NemoStation/Marlin-2B](https://huggingface.co/NemoStation/Marlin-2B) (2B video44VLM, gated — accept the license on the model page first) over a directory of videos45via vLLM. Output is a resumable parquet dataset: one row per ~60s chunk with `scene`,46`caption`, and an `events` column of `<start - end>` descriptions in seconds.47Re-running skips completed rows; failed rows are recorded, not dropped48(`--retry-errors` re-attempts them).49 50Videos longer than ~60s are split into chunks and event timestamps offset back to51global film time. This is required for correct timestamps, not an optimisation:52Marlin was trained on short clips and compresses any input onto a ~60s timeline.53 54`--find "event"` switches to grounding mode: each chunk returns a candidate55`(span_start, span_end)`. Spans are candidates, not detections — the model cannot56say "not present", so filter or verify downstream. When the event is real, spans57are precise to fractions of a second.58 59**Cost**: ~3s of GPU per minute of film on `a10g-small` at batch scale — about60$0.05 per hour of footage.61 62**Memory**: defaults encode a measured config (`--mm-processor-cache-gb 0`, in-flight63window capped at 24). vLLM's multimodal cache grows without bound on distinct videos64and will OOM a 15 GB node if re-enabled. On `a10g-large` and up, `--window-max 64`65is safe.66 67Run `--help` on the script for all options.68