shiniagarwal/rav4-semantic-video-index

RAV4 Image to Video Semantic Retrieval Dataset

This dataset contains outputs from a system that searches a video using an image of a car part.

We give the system a picture (for example a hood or door), and it returns the moments in the video where that same part appears.

The system matches meaning (car parts), not exact pixels.

How the system works

1. 1.The video is converted into frames (1 frame per second)
2. 2.A trained detector finds car parts in every frame
3. 3.All detections are stored in a searchable table
4. 4.A query image is analyzed to detect its parts
5. 5.The system finds timestamps where the same parts appear
6. 6.Nearby timestamps are grouped into watchable clips

Both intermediate and final results are provided.

Files

1. video_detections.parquet

This is the video index.

Each row represents one detected object in one frame.

Columns:

• —video_id — id of the video
• —frame_index — frame number (sampled at 1 FPS)
• —timestamp — second in the original video
• —class_label — detected car part (hood, bumper, mirror, glass, etc.)
• —xmin, ymin, xmax, ymax — bounding box location
• —confidence_score — model confidence for that detection

Low confidence detections may later be ignored during retrieval.

2. retrievalresultsraw.parquet

This file stores the first retrieval matches.

Process:

• —detect parts in query image
• —search the video index
• —merge consecutive seconds into short segments

Each row is one continuous appearance of a component.

Columns:

• —query_index — query image id
• —start_timestamp — beginning of appearance
• —end_timestamp — end of appearance
• —class_label — matched component
• —numberofsupporting_detections — number of frames inside the segment containing the object
• —youtube_url — verification link

These segments can still be small and fragmented.

3. retrievalresultsfinal.parquet

This file stores the final search results returned to the user.

We improve the raw segments using two ideas:

1. 1.Confidence filtering Detections with very low model confidence are ignored.

1. 1.Temporal consistency If the object appears in many nearby frames, we trust it more and merge segments into longer clips.

Columns:

• —query_index — query image id
• —class_labels — list of detected components used for retrieval (multiple parts may match)
• —start_timestamp — beginning of clip
• —end_timestamp — end of clip
• —numberofsupporting_detections — total detections inside the clip
• —youtube_url — direct video link

Difference between raw and final

retrieval_raw.parquet Short appearance segments directly from detection timeline

retrieval_final.parquet Longer reliable clips after:

• —removing weak detections
• —merging nearby segments

Goal

Enable semantic video search:

Given an image of a car component, return the moments in the video where that component appears.