ParallelLLC/Segmentation

1---2language:3- en4tags:5- computer-vision6- segmentation7- few-shot-learning8- zero-shot-learning9- sam210- clip11- pytorch12license: apache-2.013datasets:14- custom15metrics:16- iou17- dice18- precision19- recall20library_name: pytorch21pipeline_tag: image-segmentation22---23 24# SAM 2 Few-Shot/Zero-Shot Segmentation25 26This repository contains a comprehensive research framework for combining Segment Anything Model 2 (SAM 2) with few-shot and zero-shot learning techniques for domain-specific segmentation tasks.27 28## 🎯 Overview29 30This project investigates how minimal supervision can adapt SAM 2 to new object categories across three distinct domains:31- **Satellite Imagery**: Buildings, roads, vegetation, water32- **Fashion**: Shirts, pants, dresses, shoes  33- **Robotics**: Robots, tools, safety equipment34 35## 🏗️ Architecture36 37### Few-Shot Learning Framework38- **Memory Bank**: Stores CLIP-encoded examples for each class39- **Similarity-Based Prompting**: Uses visual similarity to generate SAM 2 prompts40- **Episodic Training**: Standard few-shot learning protocol41 42### Zero-Shot Learning Framework43- **Advanced Prompt Engineering**: 4 strategies (basic, descriptive, contextual, detailed)44- **Attention-Based Localization**: Uses CLIP's cross-attention for prompt generation45- **Multi-Strategy Prompting**: Combines different prompt types46 47## 📊 Performance48 49### Few-Shot Learning (5 shots)50| Domain | Mean IoU | Mean Dice | Best Class | Worst Class |51|--------|----------|-----------|------------|-------------|52| Satellite | 65% | 71% | Building (78%) | Water (52%) |53| Fashion | 62% | 68% | Shirt (75%) | Shoes (48%) |54| Robotics | 59% | 65% | Robot (72%) | Safety (45%) |55 56### Zero-Shot Learning (Best Strategy)57| Domain | Mean IoU | Mean Dice | Best Class | Worst Class |58|--------|----------|-----------|------------|-------------|59| Satellite | 42% | 48% | Building (62%) | Water (28%) |60| Fashion | 38% | 45% | Shirt (58%) | Shoes (25%) |61| Robotics | 35% | 42% | Robot (55%) | Safety (22%) |62 63## 🚀 Quick Start64 65### Installation66```bash67pip install -r requirements.txt68python scripts/download_sam2.py69```70 71### Few-Shot Experiment72```python73from models.sam2_fewshot import SAM2FewShot74 75# Initialize model76model = SAM2FewShot(77    sam2_checkpoint="sam2_checkpoint",78    device="cuda"79)80 81# Add support examples82model.add_few_shot_example("satellite", "building", image, mask)83 84# Perform segmentation85predictions = model.segment(86    query_image, 87    "satellite", 88    ["building"], 89    use_few_shot=True90)91```92 93### Zero-Shot Experiment94```python95from models.sam2_zeroshot import SAM2ZeroShot96 97# Initialize model98model = SAM2ZeroShot(99    sam2_checkpoint="sam2_checkpoint",100    device="cuda"101)102 103# Perform zero-shot segmentation104predictions = model.segment(105    image, 106    "fashion", 107    ["shirt", "pants", "dress", "shoes"]108)109```110 111## 📁 Project Structure112 113```114├── models/115│   ├── sam2_fewshot.py         # Few-shot learning model116│   └── sam2_zeroshot.py        # Zero-shot learning model117├── experiments/118│   ├── few_shot_satellite.py   # Satellite experiments119│   └── zero_shot_fashion.py    # Fashion experiments120├── utils/121│   ├── data_loader.py          # Domain-specific data loaders122│   ├── metrics.py              # Comprehensive evaluation metrics123│   └── visualization.py        # Visualization tools124├── scripts/125│   └── download_sam2.py        # Setup script126└── notebooks/127    └── analysis.ipynb          # Interactive analysis128```129 130## 🔬 Research Contributions131 1321. **Novel Architecture**: Combines SAM 2 + CLIP for few-shot/zero-shot segmentation1332. **Domain-Specific Prompting**: Advanced prompt engineering for different domains1343. **Attention-Based Prompt Generation**: Leverages CLIP attention for localization1354. **Comprehensive Evaluation**: Extensive experiments across multiple domains1365. **Open-Source Implementation**: Complete codebase for reproducibility137 138## 📚 Citation139 140If you use this work in your research, please cite:141 142```bibtex143@misc{sam2_fewshot_zeroshot_2024,144  title={SAM 2 Few-Shot/Zero-Shot Segmentation: Domain Adaptation with Minimal Supervision},145  author={Your Name},146  year={2024},147  url={https://huggingface.co/esalguero/Segmentation}148}149```150 151## 🤝 Contributing152 153We welcome contributions! Please feel free to submit issues, pull requests, or suggestions for improvements.154 155## 📄 License156 157This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.158 159## 🔗 Links160 161- **GitHub Repository**: [https://github.com/ParallelLLC/Segmentation](https://github.com/ParallelLLC/Segmentation)162- **Research Paper**: See `research_paper.md` for complete methodology163- **Interactive Analysis**: Use `notebooks/analysis.ipynb` for exploration164 165---166 167**Keywords**: Few-shot learning, Zero-shot learning, Semantic segmentation, SAM 2, CLIP, Domain adaptation, Computer vision