AI-Talent-Force/dev_caio

1# ShortSmith v2 - Implementation Plan2 3## Overview4Build a Hugging Face Space that extracts "hype" moments from videos with optional person-specific filtering.5 6## Project Structure7```8shortsmith-v2/9├── app.py                    # Gradio UI (Hugging Face interface)10├── requirements.txt          # Dependencies11├── config.py                 # Configuration and constants12├── utils/13│   ├── __init__.py14│   ├── logger.py             # Centralized logging15│   └── helpers.py            # Utility functions16├── core/17│   ├── __init__.py18│   ├── video_processor.py    # FFmpeg video/audio extraction19│   ├── scene_detector.py     # PySceneDetect integration20│   ├── frame_sampler.py      # Hierarchical sampling logic21│   └── clip_extractor.py     # Final clip cutting22├── models/23│   ├── __init__.py24│   ├── visual_analyzer.py    # Qwen2-VL integration25│   ├── audio_analyzer.py     # Wav2Vec 2.0 + Librosa26│   ├── face_recognizer.py    # InsightFace (SCRFD + ArcFace)27│   ├── body_recognizer.py    # OSNet for body recognition28│   ├── motion_detector.py    # RAFT optical flow29│   └── tracker.py            # ByteTrack integration30├── scoring/31│   ├── __init__.py32│   ├── hype_scorer.py        # Hype scoring logic33│   └── domain_presets.py     # Domain-specific weights34└── pipeline/35    ├── __init__.py36    └── orchestrator.py       # Main pipeline coordinator37```38 39## Implementation Phases40 41### Phase 1: Core Infrastructure421. **config.py** - Configuration management43   - Model paths, thresholds, domain presets44   - HuggingFace API key handling45 462. **utils/logger.py** - Centralized logging47   - File and console handlers48   - Different log levels per module49   - Timing decorators for performance tracking50 513. **utils/helpers.py** - Common utilities52   - File validation53   - Temporary file management54   - Error formatting55 56### Phase 2: Video Processing Layer574. **core/video_processor.py** - FFmpeg operations58   - Extract frames at specified FPS59   - Extract audio track60   - Get video metadata (duration, resolution, fps)61   - Cut clips at timestamps62 635. **core/scene_detector.py** - Scene boundary detection64   - PySceneDetect integration65   - Content-aware detection66   - Return scene timestamps67 686. **core/frame_sampler.py** - Hierarchical sampling69   - First pass: 1 frame per 5-10 seconds70   - Second pass: Dense sampling on candidates71   - Dynamic FPS based on motion72 73### Phase 3: AI Models747. **models/visual_analyzer.py** - Qwen2-VL-2B75   - Load quantized model76   - Process frame batches77   - Extract visual embeddings/scores78 798. **models/audio_analyzer.py** - Audio analysis80   - Librosa for basic features (RMS, spectral flux, centroid)81   - Optional Wav2Vec 2.0 for advanced understanding82   - Return audio hype signals per segment83 849. **models/face_recognizer.py** - Face detection/recognition85   - InsightFace SCRFD for detection86   - ArcFace for embeddings87   - Reference image matching88 8910. **models/body_recognizer.py** - Body recognition90    - OSNet for full-body embeddings91    - Handle non-frontal views92 9311. **models/motion_detector.py** - Motion analysis94    - RAFT optical flow95    - Motion magnitude scoring96 9712. **models/tracker.py** - Multi-object tracking98    - ByteTrack integration99    - Maintain identity across frames100 101### Phase 4: Scoring & Selection10213. **scoring/domain_presets.py** - Domain configurations103    - Sports, Vlogs, Music, Podcasts presets104    - Custom weight definitions105 10614. **scoring/hype_scorer.py** - Hype calculation107    - Combine visual + audio scores108    - Apply domain weights109    - Normalize and rank segments110 111### Phase 5: Pipeline & UI11215. **pipeline/orchestrator.py** - Main coordinator113    - Coordinate all components114    - Handle errors gracefully115    - Progress reporting116 11716. **app.py** - Gradio interface118    - Video upload119    - API key input (secure)120    - Prompt/instructions input121    - Domain selection122    - Reference image upload (for person filtering)123    - Progress bar124    - Output video gallery125 126## Key Design Decisions127 128### Error Handling Strategy129- Each module has try/except with specific exception types130- Errors bubble up with context131- Pipeline continues with degraded functionality when possible132- User-friendly error messages in UI133 134### Logging Strategy135- DEBUG: Model loading, frame processing details136- INFO: Pipeline stages, timing, results137- WARNING: Fallback triggers, degraded mode138- ERROR: Failures with stack traces139 140### Memory Management141- Process frames in batches142- Clear GPU memory between stages143- Use generators where possible144- Temporary file cleanup145 146### HuggingFace Space Considerations147- Use `gr.State` for session data148- Respect ZeroGPU limits (if using)149- Cache models in `/tmp` or HF cache150- Handle timeouts gracefully151 152## API Key Usage153The API key input is for future extensibility (e.g., external services).154For MVP, all processing is local using open-weight models.155 156## Gradio UI Layout157```158┌─────────────────────────────────────────────────────────────┐159│  ShortSmith v2 - AI Video Highlight Extractor               │160├─────────────────────────────────────────────────────────────┤161│  ┌─────────────────────┐  ┌─────────────────────────────┐   │162│  │ Upload Video        │  │ Settings                    │   │163│  │ [Drop zone]         │  │ Domain: [Dropdown]          │   │164│  │                     │  │ Clip Duration: [Slider]     │   │165│  └─────────────────────┘  │ Num Clips: [Slider]         │   │166│                           │ API Key: [Password field]   │   │167│  ┌─────────────────────┐  └─────────────────────────────┘   │168│  │ Reference Image     │                                    │169│  │ (Optional)          │  ┌─────────────────────────────┐   │170│  │ [Drop zone]         │  │ Additional Instructions     │   │171│  └─────────────────────┘  │ [Textbox]                   │   │172│                           └─────────────────────────────┘   │173├─────────────────────────────────────────────────────────────┤174│  [🚀 Extract Highlights]                                    │175├─────────────────────────────────────────────────────────────┤176│  Progress: [████████████░░░░░░░░] 60%                       │177│  Status: Analyzing audio...                                 │178├─────────────────────────────────────────────────────────────┤179│  Results                                                    │180│  ┌──────────┐ ┌──────────┐ ┌──────────┐                    │181│  │ Clip 1   │ │ Clip 2   │ │ Clip 3   │                    │182│  │ [Video]  │ │ [Video]  │ │ [Video]  │                    │183│  │ Score:85 │ │ Score:78 │ │ Score:72 │                    │184│  └──────────┘ └──────────┘ └──────────┘                    │185│  [Download All]                                             │186└─────────────────────────────────────────────────────────────┘187```188 189## Dependencies (requirements.txt)190```191gradio>=4.0.0192torch>=2.0.0193transformers>=4.35.0194accelerate195bitsandbytes196qwen-vl-utils197librosa>=0.10.0198soundfile199insightface200onnxruntime-gpu201opencv-python-headless202scenedetect[opencv]203numpy204pillow205tqdm206ffmpeg-python207```208 209## Implementation Order2101. config.py, utils/ (foundation)2112. core/video_processor.py (essential)2123. models/audio_analyzer.py (simpler, Librosa first)2134. core/scene_detector.py2145. core/frame_sampler.py2156. scoring/ modules2167. models/visual_analyzer.py (Qwen2-VL)2178. models/face_recognizer.py, body_recognizer.py2189. models/tracker.py, motion_detector.py21910. pipeline/orchestrator.py22011. app.py (Gradio UI)221 222## Notes223- Start with Librosa-only audio (MVP), add Wav2Vec later224- Face/body recognition is optional (triggered by reference image)225- Motion detection can be skipped in MVP for speed226- ByteTrack only needed when person filtering is enabled227