yuchenxie/EgoLoc-Contact-GRPO

EgoLoc Contact Exact-Moment Grid GRPO

This is a self-contained 3x3 image-grid dataset for GRPO training on exact contact/start localization. The numbered cells are chronological and use 1-based indices.

This dataset is used to improve a VLM's accuracy for the EgoLoc pipeline.

This dataset IS NOT shuffled. When undergoing GRPO, recommend shuffling the dataset.

3x3 grid dataset for VLM tuning on contact frame identification.

Splits

• —Training rows: 1389
• —Validation rows: 146

All variants from a video remain in one split. The images are copied byte-for-byte from the corresponding SFT dataset.

Columns

• —image: supplied automatically by the Hugging Face ImageFolder loader from file_name
• —prompt: asks for the exact event moment and a trailing {"points": [N]} response
• —task: contact
• —gt_cell: ground-truth cell in the range 1-9
• —cell_rewards: lookup table for predicted cells 1-9
• —reward_sigma: Gaussian sigma in cells
• —invalid_reward: reward for malformed, -1, multi-point, or out-of-range answers
• —video_source, video_index, gt_frame, offset, frame_indices: provenance and validation metadata

There is intentionally no supervised target column. GRPO samples completions online and looks up the parsed predicted cell in cell_rewards.

Reward

For predicted cell p and ground-truth cell g:

reward = exp(-((p - g)^2) / (2 * 2^2))

Exact predictions receive 1. For a ground truth at cell 1, rewards for cells 1 through 9 are:

[1.0000, 0.8825, 0.6065, 0.3247, 0.1353, 0.0439, 0.0111, 0.0022, 0.0003]

Only existing cells 1-9 are scored. Invalid outputs receive 0.

Other Details

• —Training videos: 176
• —Eval videos: 20 (10 DeskTIL + 10 EgoPAT3D)
• —Grid variants: 9 per video (GT frame at each cell position)
• —Training rows: 1389
• —Eval rows: 146
• —Skipped training rows (edge cases): 195
• —Skipped eval rows (edge cases): 34
• —Grid size: 3x3
• —Action: Grasping the object

train/metadata.jsonl is training-only. validation/metadata.jsonl is the held-out eval subset. This layout is compatible with Hugging Face ImageFolder split detection. Each row contains a grid image, a prompt, and a target response with the cell number (1-9) of the contact frame.

Each row contains a grid image, a prompt, and a target response with the cell number (1-9) of the contact frame.

Why skipped (edge cases)?

When the script generates 9 grid variants per video, it slides a window of 9 consecutive frames so the GT frame lands at each of the 9 positions. But some offsets are impossible if the GT frame is too close to the start or end of the video.