Rakancorle1/vtg-bench-frontier-eval
Video temporal grounding with frontier models — frame-based eval (Sept 2026) Can a frontier multimodal model, given a video and a question, return the answer and the time interval where it happens? Four models were tested with byte-identical inputs on 351 human-annotated questions over 178 YouTube videos (dashcam, SNL sketches, talk shows, vlogs, soccer and basketball highlights): Claude Fable 5.1, Claude Fable 5, GPT-6 Astra and GPT-5.6 Sol (all via Amazon Bedrock). Neither… See the full description on the dataset page: https://huggingface.co/datasets/Rakancorle1/vtg-bench-frontier-eval.
Video temporal grounding with frontier models — frame-based eval (Sept 2026)
Can a frontier multimodal model, given a video and a question, return the answer and the time interval where it happens? Four models were tested with byte-identical inputs on 351 human-annotated questions over 178 YouTube videos (dashcam, SNL sketches, talk shows, vlogs, soccer and basketball highlights): Claude Fable 5.1, Claude Fable 5, GPT-6 Astra and GPT-5.6 Sol (all via Amazon Bedrock).
Neither model family accepts video input, so each video is turned into timestamped frames: 96 uniform frames first, then a 2 fps zoom into the model's own coarse interval. Full protocol in code/README.md.
(clean subset, n = 319; answer accuracy on the 159 questions with a reference answer, judged by Claude Opus 5)
Layout
code/— the pipeline scripts, with a README describing each one and how to run them.data/— the labels (two annotation workbooks, normalisedqa_all.jsonl/qa_matched.jsonl), the Drive manifests and listings, durations. The videos and the frame cache are not in this repo: they are YouTube content and live as.tar.gzarchives in the separate media repo <https://huggingface.co/datasets/Rakancorle1/vtg-frontier-eval-media> (seedata/README.md).results/— every model's raw output for every question (both passes), the Opus 5 answer judgements, the aggregated report and a per-question spreadsheet.