danelcsb/sam2.1_hiera_tiny
11.4k
1---2library_name: transformers3license: apache-2.04pipeline_tag: image-segmentation5---6 7# Model Card for SAM 2: Segment Anything in Images and Videos8 9Repository for SAM 2: Segment Anything in Images and Videos, a foundation model towards solving promptable visual segmentation in images and videos from FAIR. See the SAM 2 paper for more information.10 1112 13## Model Details14 15### Model Description16 17SAM 2 (Segment Anything Model 2) is a foundation model developed by Meta FAIR for promptable visual segmentation across both images and videos. It extends the capabilities of the original SAM by introducing a memory-driven, streaming architecture that enables real-time, interactive segmentation and tracking of objects even as they change or temporarily disappear across video frames. SAM 2 achieves state-of-the-art segmentation accuracy with significantly improved speed and data efficiency, outperforming existing models for both images and videos.18 19This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.20 21- **Developed by:** Meta FAIR (Meta AI Research), Authors: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer.22- **Shared by [optional]:** [Sangbum Choi](https://www.linkedin.com/in/daniel-choi-86648216b/) and [Yoni Gozlan](https://huggingface.co/yonigozlan)23- **Model type:** Transformer-based promptable visual segmentation model with streaming memory module for videos.24- **License:** Apache-2.0, BSD 3-Clause25 26### Model Sources [optional]27 28<!-- Provide the basic links for the model. -->29 30- **Repository:** https://github.com/facebookresearch/sam231- **Paper [optional]:** https://arxiv.org/abs/2408.0071432- **Demo [optional]:** https://ai.meta.com/sam2/33 34## Uses35 36<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->37 38### Direct Use39 40SAM 2 is designed for:41 42Promptable segmentation—select any object in video or image using points, boxes, or masks as prompts. 43 44Zero-shot segmentation—performs strongly even on objects, image domains, or videos not seen during training.45 46Real-time, interactive applications—track or segment objects across frames, allowing corrections/refinements with new prompts as needed.47 48Research and industrial applications—facilitates precise object segmentation in video editing, robotics, AR, medical imaging, and more.49 50## Bias, Risks, and Limitations51 52Generalization Limits: While designed for zero-shot generalization, rare or unseen visual domains may challenge model reliability.53 54### Recommendations55 56Human-in-the-loop review is advised for critical use cases.57 58Users should evaluate and possibly retrain or fine-tune SAM 2 for highly specific domains.59 60Ethical and privacy considerations must be taken into account, especially in surveillance or sensitive settings.61 62## How to Get Started with the Model63 64```65from transformers import (66 Sam2Config,67 Sam2ImageProcessorFast,68 Sam2MaskDecoderConfig,69 Sam2MemoryAttentionConfig,70 Sam2MemoryEncoderConfig,71 Sam2Model,72 Sam2Processor,73 Sam2PromptEncoderConfig,74 Sam2VideoProcessor,75 Sam2VisionConfig,76)77 78image_processor = Sam2ImageProcessorFast()79video_processor = Sam2VideoProcessor()80processor = Sam2Processor(image_processor=image_processor, video_processor=video_processor)81 82sam2model = Sam2Model.from_pretrained("danelcsb/sam2.1_hiera_tiny").to("cuda")83 84# `video_dir` a directory of JPEG frames with filenames like `<frame_index>.jpg`85# Try to load your custom video in here86video_dir = "./videos/bedroom"87 88# scan all the JPEG frame names in this directory89frame_names = [90 p for p in os.listdir(video_dir)91 if os.path.splitext(p)[-1] in [".jpg", ".jpeg", ".JPG", ".JPEG"]92]93frame_names.sort(key=lambda p: int(os.path.splitext(p)[0]))94 95videos = []96for frame_name in frame_names:97 videos.append(Image.open(os.path.join(video_dir, frame_name)))98inference_state = processor.init_video_session(video=videos, inference_device="cuda")99inference_state.reset_inference_session()100 101ann_frame_idx = 0 # the frame index we interact with102ann_obj_id = 1 # give a unique id to each object we interact with (it can be any integers)103points = np.array([[210, 350]], dtype=np.float32)104# for labels, `1` means positive click and `0` means negative click105labels = np.array([1], np.int32)106 107# Let's add a positive click at (x, y) = (210, 350) to get started108inference_state = processor.process_new_points_or_box_for_video_frame(109 inference_state=inference_state,110 frame_idx=ann_frame_idx,111 obj_ids=ann_obj_id,112 input_points=points,113 input_labels=labels114)115any_res_masks, video_res_masks = sam2model.infer_on_video_frame_with_new_inputs(116 inference_state=inference_state,117 frame_idx=ann_frame_idx,118 obj_ids=ann_obj_id,119 consolidate_at_video_res=False,120)121```122 123## Training Details124 125### Training Data126 127Trained using a data engine that collected the largest known video segmentation dataset, SA-V (Segment Anything Video dataset), via interactive human-model collaboration.128 129Focused on full objects and parts, not restricted by semantic classes.130 131### Training Procedure132 133Preprocessing: Images and videos processed into masklets (spatio-temporal masks); prompts collected via human and model interaction loops.134 135Training regime: Used standard transformer training routines with enhancements for real-time processing; likely mixed precision for scaling to large datasets.136 137 138## Evaluation139 140 141### Testing Data, Factors & Metrics142 143#### Testing Data144 145Evaluated on SA-V and other standard video and image segmentation benchmarks.146 147#### Metrics148 149Segmentation accuracy (IoU, Dice). Speed/Throughput (frames per second).150 151SAM 2.1 checkpoints152 153The table below shows the improved SAM 2.1 checkpoints released on September 29, 2024.154| **Model** | **Size (M)** | **Speed (FPS)** | **SA-V test (J&F)** | **MOSE val (J&F)** | **LVOS v2 (J&F)** |155| :------------------: | :----------: | :--------------------: | :-----------------: | :----------------: | :---------------: |156| sam2.1_hiera_tiny | 38.9 | 91.2 | 76.5 | 71.8 | 77.3 |157| sam2.1_hiera_small | 46 | 84.8 | 76.6 | 73.5 | 78.3 |158| sam2.1_hiera_base_plus| 80.8 | 64.1 | 78.2 | 73.7 | 78.2 |159| sam2.1_hiera_large | 224.4 | 39.5 | 79.5 | 74.6 | 80.6 |160 161SAM 2 checkpoints162 163The previous SAM 2 checkpoints released on July 29, 2024 can be found as follows:164 165| **Model** | **Size (M)** | **Speed (FPS)** | **SA-V test (J&F)** | **MOSE val (J&F)** | **LVOS v2 (J&F)** |166| :------------------: | :----------: | :--------------------: | :-----------------: | :----------------: | :---------------: |167| sam2_hiera_tiny | 38.9 | 91.5 | 75.0 | 70.9 | 75.3 |168| sam2_hiera_small | 46 | 85.6 | 74.9 | 71.5 | 76.4 |169| sam2_hiera_base_plus | 80.8 | 64.8 | 74.7 | 72.8 | 75.8 |170| sam2_hiera_large | 224.4 | 39.7 | 76.0 | 74.6 | 79.8 |171 172 173### Results174 175Video segmentation: Higher accuracy with 3x fewer user prompts versus prior approaches.176 177Image segmentation: 6x faster and more accurate than original SAM.178 179## Citation [optional]180 181**BibTeX:**182 183@article{ravi2024sam2,184 title={SAM 2: Segment Anything in Images and Videos},185 author={Nikhila Ravi and Valentin Gabeur and Yuan-Ting Hu and Ronghang Hu and Chaitanya Ryali and Tengyu Ma and Haitham Khedr and Roman R{\"a}dle and Chloe Rolland and Laura Gustafson and Eric Mintun and Junting Pan and Kalyan Vasudev Alwala and Nicolas Carion and Chao-Yuan Wu and Ross Girshick and Piotr Doll\'ar and Christoph Feichtenhofer},186 journal={arXiv preprint arXiv:2408.00714},187 year={2024}188}189 190**APA:**191 192Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., & Feichtenhofer, C. (2024). SAM 2: Segment Anything in Images and Videos. arXiv preprint arXiv:2408.00714.193 194## Model Card Authors [optional]195 196[Sangbum Choi](https://www.linkedin.com/in/daniel-choi-86648216b/) and [Yoni Gozlan](https://huggingface.co/yonigozlan)197 198## Model Card Contact199 200Meta FAIR (contact via support@segment-anything.com)