userdarius/eval3_track3_vl_pairs

eval3track3vl_pairs (v3 — image↔label mispairing fixed)

ObjectVLA-style vision-language co-training data for the Track 3 TOY 3-celeb Eval 3 task. v3 fixes a critical bug where ~98% of rows had image_path pointing to a frame extracted from a different episode than the row's labels described.

Companion to `HBOrtiz/so101_eval3_track3_v3_baseline`.

What changed vs prior pushes

Bug fix (v3): Earlier builders keyed video lookups by enumeration index in their own filtered epmetadata list, but `eptovideos` was keyed by the merged-dataset's `episodeindex (a different index space — assigned by mergetrack3custom.discover() = sorted(base) + sorted(aug)`). The builder filter selected 151 base teleops vs the merger's 178, shifting every subsequent index. Result: ~98% of rows mispaired.

v3 replicates merge_track3_custom.discover() exactly to produce the canonical episode-name list (length 9394 = merger's episode_index order), then joins per-episode metadata by NAME. Result: every JPEG is now extracted from the episode whose labels the row carries.

The 27 base teleops without portrait_corners.json caches are skipped (no usable bbox geometry available for them).

Contents

• —`manifest.parquet` — 56 202 VL pair rows (9 367 episodes × 3 portraits × 2 caption types).
• —`data.tar.zst` — images/chunk-*/<episode>__f0000.jpg + references/<episode>__ref.jpg. Extract with tar --zstd -xf data.tar.zst.
• —`_stats.json` — counts + refit success rate.

Schema

Caption types

`location_explicit` — prompt="What is in this image?", target="The printed photo of <name> is at [x1,y1,x2,y2,x3,y3,x4,y4]." `qa_grounded` — prompt="Who is in the printed photo at [x1,y1,x2,y2,x3,y3,x4,y4]?", target="<name>"

The 8 numbers in the prompts/targets are the flat list of the 4 quad corners (preserves rotation, unlike axis-aligned xyxy).

Stats

• —3 unique celebs: Yann LeCun, Barack Obama, Taylor Swift
• —9 367 episodes (178 base + 9 216 aug − 27 cacheless base)
• —56 202 VL pairs
• —78.9 % bbox_refit_ok=True (sub-pixel refined corners)
• —21.1 % bbox_refit_ok=False (coarse minAreaRect fallback — still 4 distinct corners)

Loading

import subprocess
from huggingface_hub import hf_hub_download
import pyarrow.parquet as pq

tar = hf_hub_download("HBOrtiz/eval3_track3_vl_pairs", "data.tar.zst", repo_type="dataset")
subprocess.run(["tar", "--zstd", "-xf", tar])

manifest = pq.read_table(hf_hub_download(
    "HBOrtiz/eval3_track3_vl_pairs", "manifest.parquet", repo_type="dataset"
)).to_pandas()