David310/SAM2_video_mask_generator
3
1# Segment Anything Video (SA-V) Dataset2 3## Overview4 5[Segment Anything Video (SA-V)](https://ai.meta.com/datasets/segment-anything-video/), consists of 51K diverse videos and 643K high-quality spatio-temporal segmentation masks (i.e., masklets). The dataset is released under the CC by 4.0 license. Browse the dataset [here](https://sam2.metademolab.com/dataset).6 78 9## Getting Started10 11### Download the dataset12 13Visit [here](https://ai.meta.com/datasets/segment-anything-video-downloads/) to download SA-V including the training, val and test sets.14 15### Dataset Stats16 17| | Num Videos | Num Masklets |18| ---------- | ---------- | ----------------------------------------- |19| SA-V train | 50,583 | 642,036 (auto 451,720 and manual 190,316) |20| SA-V val | 155 | 293 |21| SA-V test | 150 | 278 |22 23### Notebooks24 25To load and visualize the SA-V training set annotations, refer to the example [sav_visualization_example.ipynb](./sav_visualization_example.ipynb) notebook.26 27### SA-V train28 29For SA-V training set we release the mp4 videos and store the masklet annotations per video as json files . Automatic masklets and manual masklets are stored separately as two json files: `{video_id}_auto.json` and `{video_id}_manual.json`. They can be loaded as dictionaries in python in the format below.30 31```32{33 "video_id" : str; video id34 "video_duration" : float64; the duration in seconds of this video35 "video_frame_count" : float64; the number of frames in the video36 "video_height" : float64; the height of the video37 "video_width" : float64; the width of the video38 "video_resolution" : float64; video_height $\times$ video_width39 "video_environment" : List[str]; "Indoor" or "Outdoor"40 "video_split" : str; "train" for training set41 "masklet" : List[List[Dict]]; masklet annotations in list of list of RLEs.42 The outer list is over frames in the video and the inner list43 is over objects in the video.44 "masklet_id" : List[int]; the masklet ids45 "masklet_size_rel" : List[float]; the average mask area normalized by resolution46 across all the frames where the object is visible47 "masklet_size_abs" : List[float]; the average mask area (in pixels)48 across all the frames where the object is visible49 "masklet_size_bucket" : List[str]; "small": $1$ <= masklet_size_abs < $32^2$,50 "medium": $32^2$ <= masklet_size_abs < $96^2$,51 and "large": masklet_size_abs > $96^2$52 "masklet_visibility_changes" : List[int]; the number of times where the visibility changes53 after the first appearance (e.g., invisible -> visible54 or visible -> invisible)55 "masklet_first_appeared_frame" : List[int]; the index of the frame where the object appears56 the first time in the video. Always 0 for auto masklets.57 "masklet_frame_count" : List[int]; the number of frames being annotated. Note that58 videos are annotated at 6 fps (annotated every 4 frames)59 while the videos are at 24 fps.60 "masklet_edited_frame_count" : List[int]; the number of frames being edited by human annotators.61 Always 0 for auto masklets.62 "masklet_type" : List[str]; "auto" or "manual"63 "masklet_stability_score" : Optional[List[List[float]]]; per-mask stability scores. Auto annotation only.64 "masklet_num" : int; the number of manual/auto masklets in the video65 66}67```68 69Note that in SA-V train, there are in total 50,583 videos where all of them have manual annotations. Among the 50,583 videos there are 48,436 videos that also have automatic annotations.70 71### SA-V val and test72 73For SA-V val and test sets, we release the extracted frames as jpeg files, and the masks as png files with the following directory structure:74 75```76sav_val(sav_test)77├── sav_val.txt (sav_test.txt): a list of video ids in the split78├── JPEGImages_24fps # videos are extracted at 24 fps79│ ├── {video_id}80│ │ ├── 00000.jpg # video frame81│ │ ├── 00001.jpg # video frame82│ │ ├── 00002.jpg # video frame83│ │ ├── 00003.jpg # video frame84│ │ └── ...85│ ├── {video_id}86│ ├── {video_id}87│ └── ...88└── Annotations_6fps # videos are annotated at 6 fps89 ├── {video_id}90 │ ├── 000 # obj 00091 │ │ ├── 00000.png # mask for object 000 in 00000.jpg92 │ │ ├── 00004.png # mask for object 000 in 00004.jpg93 │ │ ├── 00008.png # mask for object 000 in 00008.jpg94 │ │ ├── 00012.png # mask for object 000 in 00012.jpg95 │ │ └── ...96 │ ├── 001 # obj 00197 │ ├── 002 # obj 00298 │ └── ...99 ├── {video_id}100 ├── {video_id}101 └── ...102```103 104All masklets in val and test sets are manually annotated in every frame by annotators. For each annotated object in a video, we store the annotated masks in a single png. This is because the annotated objects may overlap, e.g., it is possible in our SA-V dataset for there to be a mask for the whole person as well as a separate mask for their hands.105 106## SA-V Val and Test Evaluation107 108We provide an evaluator to compute the common J and F metrics on SA-V val and test sets. To run the evaluation, we need to first install a few dependencies as follows:109 110```111pip install -r requirements.txt112```113 114Then we can evaluate the predictions as follows:115 116```117python sav_evaluator.py --gt_root {GT_ROOT} --pred_root {PRED_ROOT}118```119 120or run121 122```123python sav_evaluator.py --help124```125 126to print a complete help message.127 128The evaluator expects the `GT_ROOT` to be one of the following folder structures, and `GT_ROOT` and `PRED_ROOT` to have the same structure.129 130- Same as SA-V val and test directory structure131 132```133{GT_ROOT} # gt root folder134├── {video_id}135│ ├── 000 # all masks associated with obj 000136│ │ ├── 00000.png # mask for object 000 in frame 00000 (binary mask)137│ │ └── ...138│ ├── 001 # all masks associated with obj 001139│ ├── 002 # all masks associated with obj 002140│ └── ...141├── {video_id}142├── {video_id}143└── ...144```145 146In the paper for the experiments on SA-V val and test, we run inference on the 24 fps videos, and evaluate on the subset of frames where we have ground truth annotations (first and last annotated frames dropped). The evaluator will ignore the masks in frames where we don't have ground truth annotations.147 148- Same as [DAVIS](https://github.com/davisvideochallenge/davis2017-evaluation) directory structure149 150```151{GT_ROOT} # gt root folder152├── {video_id}153│ ├── 00000.png # annotations in frame 00000 (may contain multiple objects)154│ └── ...155├── {video_id}156├── {video_id}157└── ...158```159 160## License161 162The evaluation code is licensed under the [BSD 3 license](./LICENSE). Please refer to the paper for more details on the models. The videos and annotations in SA-V Dataset are released under CC BY 4.0.163 164Third-party code: the evaluation software is heavily adapted from [`VOS-Benchmark`](https://github.com/hkchengrex/vos-benchmark) and [`DAVIS`](https://github.com/davisvideochallenge/davis2017-evaluation) (with their licenses in [`LICENSE_DAVIS`](./LICENSE_DAVIS) and [`LICENSE_VOS_BENCHMARK`](./LICENSE_VOS_BENCHMARK)).165