facebook/cotracker3

Point tracking with CoTracker3

CoTracker3 is a fast transformer-based model that was introduced in CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. It can track any point in a video and brings to tracking some of the benefits of Optical Flow. You could read more about the paper on our webpage. Code is available here.

CoTracker can track:

• —Any pixel in a video
• —A quasi-dense set of pixels together
• —Points can be manually selected or sampled on a grid in any video frame

How to use

Here is how to use this model in the offline mode:

import torch

Download the video

url = 'https://github.com/facebookresearch/co-tracker/raw/refs/heads/main/assets/apple.mp4'

import imageio.v3 as iio frames = iio.imread(url, plugin="FFMPEG") # plugin="pyav"

device = 'cuda' grid_size = 10 video = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device) # B T C H W

Run Offline CoTracker:

cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker3offline").to(device) predtracks, predvisibility = cotracker(video, gridsize=grid_size) # B T N 2, B T N 1

and in the **online mode**:

cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker3_online").to(device)

Run Online CoTracker, the same model with a different API:

Initialize online processing

cotracker(videochunk=video, isfirststep=True, gridsize=grid_size)

Process the video

for ind in range(0, video.shape[1] - cotracker.step, cotracker.step): predtracks, predvisibility = cotracker( video_chunk=video[:, ind : ind + cotracker.step * 2] ) # B T N 2, B T N 1

Online processing is more memory-efficient and allows for the processing of longer videos or videos in real-time.

## BibTeX entry and citation info

@inproceedings{karaev24cotracker3, title = {CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos}, author = {Nikita Karaev and Iurii Makarov and Jianyuan Wang and Natalia Neverova and Andrea Vedaldi and Christian Rupprecht}, booktitle = {Proc. {arXiv:2410.11831}}, year = {2024} }