ary22/rav4-semantic-search

RAV4 Video Semantic Search — DS681 Assignment 2

Goal

This project builds an image-based semantic search system over a video.

Given a query image of a car component (wheel, door, mirror, etc.), the system retrieves video time intervals where the same component appears.

Video: https://www.youtube.com/watch?v=YcvECxtXoxQ

Method Overview

Pipeline:

1) Sample frames from video 2) Detect exterior car parts in each frame 3) Store detections in a structured Parquet index 4) Detect component in query image 5) Retrieve contiguous time intervals where component appears

No manual labeling or hard-coded timestamps were used.

Detector

Model: YOLOv8n-seg Dataset: Ultralytics carparts-seg (23 classes)

Detected components include:

backbumper, backdoor, backglass, backleftdoor, backleftlight, backlight, backrightdoor, backrightlight, frontbumper, frontdoor, frontglass, frontleftdoor, frontleftlight, frontlight, frontrightdoor, frontrightlight, hood, leftmirror, rightmirror, tailgate, trunk, wheel

The model was trained for 10 epochs at 640 resolution.

Video Processing

Frames sampled every 5 seconds using ffmpeg.

Each frame → detector → bounding boxes + labels + confidence

All detections stored in:

detections.parquet

Each row represents one detection.

Parquet Schema

The Parquet file is the index used for retrieval.

Query Dataset

Dataset: aegean-ai/rav4-exterior-images

For each query image: 1) Run detector 2) Identify top components 3) Find matching timestamps in video index 4) Merge contiguous detections into segments

Retrieval Output

Stored in:

retrieval_results.jsonl

Each row:

Example Verification

Wheel example:

https://www.youtube.com/embed/YcvECxtXoxQ?start=120&end=165

The retrieved clip shows the component detected by the query image.

Matching Logic

Retrieval uses semantic structure:

query image → detected class → match same class in video index

Segments formed by merging timestamps within 5 seconds.

No query-specific heuristics were used.

Limitations

• —Small parts sometimes missed at distance
• —Occlusions reduce confidence
• —Similar parts (left vs right mirror) may merge

Reproducibility

All results can be reproduced using only the Parquet detection index. The retrieval stage does not depend on manual annotations.