AI-Talent-Force/dev_caio

1# ShortSmith v2 - Testing & Evaluation Guide2 3## Overview4This document outlines our testing strategy for evaluating ShortSmith v2's highlight extraction quality. It covers what to test, how to measure results, and how to score/store findings.5 6---7 8## 1. What We're Testing9 10### 1.1 Core Quality Metrics11 12| Component | What It Does | Key Output |13|-----------|--------------|------------|14| **Visual Analysis** | Qwen2-VL rates frame excitement | hype_score (0-1), action, emotion |15| **Audio Analysis** | Librosa detects energy/beats | energy_score, excitement_score (0-1) |16| **Motion Detection** | RAFT optical flow measures action | magnitude (0-1), is_action flag |17| **Viral Hooks** | Finds optimal clip start points | hook_type, confidence, intensity |18| **Hype Scoring** | Combines all signals | combined_score (0-1), rank |19 20### 1.2 End-to-End Quality21 22- **Clip Relevance**: Are the extracted clips actually highlights?23- **Hook Effectiveness**: Do clips start at engaging moments?24- **Domain Accuracy**: Does sports mode pick different clips than podcast mode?25- **Person Filtering**: When enabled, does target person appear in clips?26 27---28 29## 2. Test Dataset Requirements30 31### 2.1 Video Categories (Minimum 3 videos per category)32 33| Domain | Video Type | Duration | Source Examples |34|--------|-----------|----------|-----------------|35| **Sports** | Football/Basketball highlights | 5-15 min | YouTube sports channels |36| **Music** | Music videos, concerts | 3-8 min | Official MVs, live performances |37| **Gaming** | Gameplay with commentary | 10-20 min | Twitch clips, YouTube gaming |38| **Vlogs** | Personal vlogs with reactions | 8-15 min | YouTube vlogs |39| **Podcasts** | Interview/discussion clips | 15-30 min | Podcast video episodes |40| **General** | Mixed content | 5-15 min | Random viral videos |41 42### 2.2 Test Video Naming Convention43```44{domain}_{video_id}_{duration_mins}.mp445 46Examples:47sports_001_10min.mp448music_002_5min.mp449gaming_003_15min.mp450```51 52---53 54## 3. Testing Methodology55 56### 3.1 Automated Metrics (System-Generated)57 58For each test run, capture these from the pipeline output:59 60```61Processing Metrics:62- processing_time_seconds63- frames_analyzed64- scenes_detected65- hooks_detected66 67Per Clip:68- clip_id (1, 2, 3)69- start_time70- end_time71- hype_score72- visual_score73- audio_score74- motion_score75- hook_type (if any)76- hook_confidence77```78 79### 3.2 Human Evaluation Criteria80 81Each clip needs human scoring on these dimensions:82 83#### A. Highlight Quality (1-5 scale)84```851 = Not a highlight at all (boring/irrelevant)862 = Weak highlight (somewhat interesting)873 = Decent highlight (would watch)884 = Good highlight (engaging)895 = Perfect highlight (would share/rewatch)90```91 92#### B. Hook Effectiveness (1-5 scale)93```941 = Starts at wrong moment (confusing/boring start)952 = Starts too early/late (misses the peak)963 = Acceptable start (gets the point across)974 = Good start (grabs attention)985 = Perfect start (immediately engaging, viral potential)99```100 101#### C. Domain Appropriateness (Yes/No/Partial)102```103Does this clip match what you'd expect for this content type?104- Sports: Action/celebration/crowd moment?105- Music: Beat drop/chorus/dance peak?106- Gaming: Clutch play/reaction/funny moment?107- Vlogs: Emotional/funny/reveal moment?108- Podcasts: Hot take/laugh/interesting point?109```110 111### 3.3 A/B Testing (Optional)112 113Compare outputs with different settings:114- Viral hooks ON vs OFF115- Different domain presets for same video116- Custom prompt vs no prompt117 118---119 120## 4. Test Execution Process121 122### 4.1 Per-Video Test Run123 124```1251. Upload video to ShortSmith1262. Select appropriate domain1273. Set: 3 clips, 15 seconds each1284. Run extraction1295. Download all 3 clips1306. Record automated metrics from log1317. Have 2+ team members score each clip1328. Calculate average scores1339. Log results in spreadsheet134```135 136### 4.2 Test Run Naming137```138{date}_{domain}_{video_id}_{tester_initials}139 140Example: 2024-01-15_sports_001_CM141```142 143---144 145## 5. Scoring & Storage146 147### 5.1 Results Spreadsheet Structure148 149Create a Google Sheet / Excel with these columns:150 151| Date | Video_ID | Domain | Clip# | Start | End | Hype_Score | Visual | Audio | Motion | Hook_Type | Human_Highlight (1-5) | Human_Hook (1-5) | Domain_Match (Y/N/P) | Notes |152|------|----------|--------|-------|-------|-----|------------|--------|-------|--------|-----------|----------------------|------------------|---------------------|-------|153 154### 5.2 Aggregate Metrics to Track155 156Calculate weekly/monthly:157 158```159Overall Quality:160- Avg Human Highlight Score (target: >3.5)161- Avg Human Hook Score (target: >3.5)162- Domain Match Rate (target: >80%)163 164Per Domain:165- Avg scores broken down by sports/music/gaming/vlogs/podcasts166 167Correlation Analysis:168- System hype_score vs Human rating correlation169- Hook confidence vs Human hook rating correlation170```171 172### 5.3 Quality Thresholds173 174| Metric | Poor | Acceptable | Good | Excellent |175|--------|------|------------|------|-----------|176| Highlight Score | <2.5 | 2.5-3.5 | 3.5-4.2 | >4.2 |177| Hook Score | <2.5 | 2.5-3.5 | 3.5-4.2 | >4.2 |178| Domain Match | <60% | 60-75% | 75-90% | >90% |179| Hype-Human Correlation | <0.3 | 0.3-0.5 | 0.5-0.7 | >0.7 |180 181---182 183## 6. Issue Tracking184 185### 6.1 Common Issues to Watch For186 187| Issue | How to Identify | Priority |188|-------|-----------------|----------|189| Wrong clip selected | Human score <2, but hype_score >0.7 | High |190| Bad hook timing | Good highlight but hook score <2 | High |191| Domain mismatch | Domain_Match = N for >30% clips | Medium |192| Person filter fail | Target person not in clips when filter enabled | Medium |193| Processing errors | Pipeline fails/crashes | Critical |194 195### 6.2 Bug Report Template196 197```198Video: {video_id}199Domain: {domain}200Issue Type: [Wrong Clip / Bad Hook / Domain Mismatch / Other]201Clip #: {1/2/3}202Timestamp: {start}-{end}203Expected: {what should have been selected}204Actual: {what was selected}205System Scores: hype={}, visual={}, audio={}, motion={}206Human Scores: highlight={}, hook={}207Notes: {additional context}208```209 210---211 212## 7. Test Schedule213 214tentative215 216## 8. Success Criteria217 218### MVP Launch Requirements219- [ ] Avg Highlight Score > 3.5 across all domains220- [ ] Avg Hook Score > 3.5 across all domains221- [ ] Domain Match Rate > 75%222- [ ] No critical processing errors223- [ ] Processing time < 10 min for 10-min video on A10G224 225### Stretch Goals226- [ ] Avg Highlight Score > 4.0227- [ ] Avg Hook Score > 4.0228- [ ] Domain Match Rate > 90%229- [ ] Hype-Human correlation > 0.6230 231---232 233## 11. Experimentation Guide234 235This section explains how to systematically test different settings to find optimal configurations.236 237### 11.1 What Settings Can Be Changed238 239| Setting | Location | Default | Range | Impact |240|---------|----------|---------|-------|--------|241| **Coarse Sample Interval** | config.py | 5.0 sec | 2-10 sec | More frames = better accuracy, slower processing |242| **Clip Duration** | UI Slider | 15 sec | 5-30 sec | Longer clips = more context, harder to keep attention |243| **Number of Clips** | UI Slider | 3 | 1-3 | More clips = more variety, possibly lower avg quality |244| **Domain Selection** | UI Dropdown | General | 6 options | Changes weight distribution for scoring |245| **Hype Threshold** | config.py | 0.3 | 0.1-0.6 | Higher = stricter filtering, fewer candidates |246| **Min Gap Between Clips** | config.py | 30 sec | 10-60 sec | Higher = more spread out clips |247| **Scene Threshold** | config.py | 27.0 | 20-35 | Lower = more scene cuts detected |248 249### 11.2 Experiment Types250 251#### Experiment A: Domain Preset Comparison252**Goal**: Verify that domain selection actually affects results253 254**Method**:2551. Pick 1 video that could fit multiple domains (e.g., a gaming video with music)2562. Run ShortSmith 3 times with different domain settings:257   - Run 1: Gaming258   - Run 2: Music259   - Run 3: General2603. Compare the 3 clips from each run261 262**What to Record**:263```264Video: {video_id}265Experiment: Domain Comparison266 267| Run | Domain | Clip1 Start | Clip2 Start | Clip3 Start | Best Clip# | Notes |268|-----|--------|-------------|-------------|-------------|------------|-------|269| 1   | Gaming |             |             |             |            |       |270| 2   | Music  |             |             |             |            |       |271| 3   | General|             |             |             |            |       |272 273Question: Did different domains pick different moments?  Y / N274Question: Which domain worked best for this video? ____________275```276 277#### Experiment B: Clip Duration Impact278**Goal**: Find optimal clip length for engagement279 280**Method**:2811. Pick 1 video2822. Run ShortSmith 3 times with different durations:283   - Run 1: 10 seconds284   - Run 2: 15 seconds285   - Run 3: 25 seconds2863. Watch all clips and score287 288**What to Record**:289```290Video: {video_id}291Experiment: Duration Comparison292 293| Duration | Clip1 Hook Score | Clip1 "Feels Complete?" | Clip1 "Too Long?" |294|----------|------------------|-------------------------|-------------------|295| 10 sec   |                  | Y/N                     | Y/N               |296| 15 sec   |                  | Y/N                     | Y/N               |297| 25 sec   |                  | Y/N                     | Y/N               |298 299Optimal duration for this content type: ___ seconds300```301 302#### Experiment C: Custom Prompt Effectiveness303**Goal**: Test if custom prompts improve clip selection304 305**Method**:3061. Pick 1 video with a specific element you want to capture (e.g., "crowd reactions")3072. Run twice:308   - Run 1: No custom prompt309   - Run 2: Custom prompt = "Focus on crowd reactions"3103. Compare results311 312**What to Record**:313```314Video: {video_id}315Custom Prompt: "________________________"316Experiment: Custom Prompt Test317 318| Run | Has Prompt? | Clip1 Has Target Element? | Clip2 Has Target? | Clip3 Has Target? |319|-----|-------------|---------------------------|-------------------|-------------------|320| 1   | No          | Y/N                       | Y/N               | Y/N               |321| 2   | Yes         | Y/N                       | Y/N               | Y/N               |322 323Did custom prompt help? Y / N / Unclear324```325 326#### Experiment D: Person Filter Accuracy327**Goal**: Test if person filtering actually prioritizes target person328 329**Method**:3301. Pick a video with 2+ people clearly visible3312. Get a clear reference photo of 1 person3323. Run twice:333   - Run 1: No reference image334   - Run 2: With reference image3354. Count target person screen time in each clip336 337**What to Record**:338```339Video: {video_id}340Target Person: {description}341Experiment: Person Filter Test342 343| Run | Filter On? | Clip1 Person % | Clip2 Person % | Clip3 Person % | Avg % |344|-----|------------|----------------|----------------|----------------|-------|345| 1   | No         |                |                |                |       |346| 2   | Yes        |                |                |                |       |347 348Did filter increase target person screen time? Y / N349By how much? ___% improvement350```351 352### 11.3 How to Document Experiments353 354#### Step 1: Create Experiment Log355Make a new sheet/tab called "Experiments" with columns:356```357| Date | Experiment_Type | Video_ID | Variable_Tested | Setting_A | Setting_B | Winner | Notes |358```359 360#### Step 2: Record Before & After361Always record:362- What you changed (the variable)363- What stayed the same (controls)364- The measurable outcome365 366#### Step 3: Take Screenshots367For each experiment:368- Screenshot the settings used369- Screenshot the processing log370- Save the output clips with clear naming:371  ```372  exp_{type}_{video}_{setting}.mp4373 374  Examples:375  exp_domain_sports001_gaming.mp4376  exp_domain_sports001_music.mp4377  exp_duration_vlog002_10sec.mp4378  exp_duration_vlog002_25sec.mp4379  ```380 381### 11.4 Recommended Experiment Order382 383**Week 1: Baseline + Domain Testing**384```385Day 1-2: Run all 6 domains on 2 "neutral" videos386         Record which domain performs best387 388Day 3-4: Run domain-matched tests389         (Sports video with Sports setting, Music with Music, etc.)390         Record scores391 392Day 5: Analyze - Do domain presets actually help vs General?393```394 395**Week 2: Duration & Clip Count Testing**396```397Day 1-2: Test 10s vs 15s vs 20s vs 25s clips398         On 3 different video types399 400Day 3-4: Test 1 clip vs 2 clips vs 3 clips401         Does quality drop with more clips?402 403Day 5: Analyze - Find optimal duration per domain404```405 406**Week 3: Advanced Features Testing**407```408Day 1-2: Custom prompt experiments409         Try 5 different prompts on same video410 411Day 3-4: Person filter experiments412         Test accuracy on 3 multi-person videos413 414Day 5: Analyze - Document which features add value415```416 417### 11.5 Experiment Scoring Template418 419For each experiment, fill out:420 421```422=== EXPERIMENT REPORT ===423 424Date: ____________425Tester: ____________426Experiment Type: [Domain / Duration / Custom Prompt / Person Filter / Other]427 428VIDEO DETAILS429- Video ID: ____________430- Domain: ____________431- Duration: ____________432- Content Description: ____________433 434WHAT I TESTED435- Variable Changed: ____________436- Setting A: ____________437- Setting B: ____________438- Setting C (if any): ____________439 440RESULTS441 442Setting A Output:443- Clip 1: Start=___ End=___ Highlight Score=___ Hook Score=___444- Clip 2: Start=___ End=___ Highlight Score=___ Hook Score=___445- Clip 3: Start=___ End=___ Highlight Score=___ Hook Score=___446- Overall Quality (1-5): ___447 448Setting B Output:449- Clip 1: Start=___ End=___ Highlight Score=___ Hook Score=___450- Clip 2: Start=___ End=___ Highlight Score=___ Hook Score=___451- Clip 3: Start=___ End=___ Highlight Score=___ Hook Score=___452- Overall Quality (1-5): ___453 454WINNER: Setting ___455 456WHY?457________________________________458________________________________459 460RECOMMENDATION:461________________________________462________________________________463```464 465### 11.6 Key Questions to Answer Through Experiments466 467After completing experiments, you should be able to answer:468 469**Domain Settings:**470- [ ] Does Sports mode actually pick action moments better than General?471- [ ] Does Music mode find beat drops better?472- [ ] Does Podcast mode find speaking highlights?473- [ ] Which domain works best for "general" content?474 475**Clip Settings:**476- [ ] What's the ideal clip duration for TikTok/Reels (vertical short-form)?477- [ ] What's the ideal clip duration for YouTube Shorts?478- [ ] Does requesting 1 clip give higher quality than requesting 3?479 480**Features:**481- [ ] Do custom prompts actually improve results?482- [ ] How accurate is person filtering? (% of clips with target person)483- [ ] What types of custom prompts work best?484 485**Quality Patterns:**486- [ ] Which content types get the best results?487- [ ] Which content types struggle?488- [ ] Are there patterns in "bad" clips? (e.g., always too early, always misses peak)489 490---491 492## Appendix A: Quick Evaluation Form493 494```495=== ShortSmith Clip Evaluation ===496 497Video ID: ____________498Domain: ____________499Date: ____________500Evaluator: ____________501 502CLIP 1 (___:___ - ___:___)503System Hype Score: ___504Highlight Quality (1-5): ___505Hook Effectiveness (1-5): ___506Domain Match (Y/N/P): ___507Notes: ________________________________508 509CLIP 2 (___:___ - ___:___)510System Hype Score: ___511Highlight Quality (1-5): ___512Hook Effectiveness (1-5): ___513Domain Match (Y/N/P): ___514Notes: ________________________________515 516CLIP 3 (___:___ - ___:___)517System Hype Score: ___518Highlight Quality (1-5): ___519Hook Effectiveness (1-5): ___520Domain Match (Y/N/P): ___521Notes: ________________________________522 523Overall Comments:524________________________________525________________________________526```527 528---529 530## Appendix B: Domain-Specific Hook Types Reference531 532### Sports533| Hook Type | What to Look For |534|-----------|-----------------|535| GOAL_MOMENT | Scoring play, basket, touchdown |536| CROWD_ERUPTION | Audience going wild |537| COMMENTATOR_HYPE | Excited commentary voice |538| REPLAY_WORTHY | Impressive athletic move |539 540### Music541| Hook Type | What to Look For |542|-----------|-----------------|543| BEAT_DROP | Bass drop, beat switch |544| CHORUS_HIT | Chorus/hook starts |545| DANCE_PEAK | Peak choreography moment |546| VISUAL_CLIMAX | Visual spectacle |547 548### Gaming549| Hook Type | What to Look For |550|-----------|-----------------|551| CLUTCH_PLAY | Skillful play under pressure |552| ELIMINATION | Kill/win moment |553| RAGE_REACTION | Streamer emotional reaction |554| UNEXPECTED | Surprise/plot twist |555 556### Vlogs557| Hook Type | What to Look For |558|-----------|-----------------|559| REVEAL | Surprise reveal |560| PUNCHLINE | Joke landing |561| EMOTIONAL_MOMENT | Tears/joy/shock |562| CONFRONTATION | Drama/tension |563 564### Podcasts565| Hook Type | What to Look For |566|-----------|-----------------|567| HOT_TAKE | Controversial opinion |568| BIG_LAUGH | Group laughter |569| REVELATION | Surprising information |570| HEATED_DEBATE | Passionate argument |571 572---573 574## Appendix C: Sample Test Results Format575 576```csv577date,video_id,domain,clip_num,start_time,end_time,hype_score,visual_score,audio_score,motion_score,hook_type,human_highlight,human_hook,domain_match,notes5782024-01-15,sports_001,sports,1,45.2,60.2,0.82,0.75,0.88,0.71,GOAL_MOMENT,5,5,Y,Perfect goal celebration5792024-01-15,sports_001,sports,2,120.5,135.5,0.71,0.68,0.74,0.65,CROWD_ERUPTION,4,4,Y,Good crowd reaction5802024-01-15,sports_001,sports,3,200.0,215.0,0.65,0.60,0.70,0.55,PEAK_ENERGY,3,3,P,Decent but not the best moment581```582