ThreadAbort/IndexTTS-Rust

1# IndexTTS-Rust Context2 3This file preserves important context for conversation continuity between Hue and Aye sessions.4 5**Last Updated:** 2025-11-166 7---8 9## The Vision10 11IndexTTS-Rust is part of a larger audio intelligence ecosystem at 8b.is:12 131. **kokoro-tiny** - Lightweight TTS (82M params, 50+ voices, on crates.io!)142. **IndexTTS-Rust** - Advanced zero-shot TTS with emotion control153. **Phoenix-Protocol** - Audio restoration/enhancement layer164. **MEM|8** - Contextual memory system (mem-8.com, mem8)17 18Together these form a complete audio intelligence pipeline.19 20---21 22## Phoenix Protocol Integration Opportunities23 24The Phoenix Protocol (phoenix-protocol/) is a PERFECT complement to IndexTTS-Rust:25 26### Direct Module Mappings27 28| Phoenix Module | IndexTTS Use Case |29|----------------|-------------------|30| `emotional.rs` | Map to our 8D emotion control (Warmth→body, Presence→power, Clarity→articulation, Air→space, Ultrasonics→depth) |31| `voice_signature.rs` | Enhance speaker embeddings for voice cloning |32| `spectral_velocity.rs` | Add momentum tracking to mel-spectrogram |33| `marine.rs` | Validate TTS output authenticity/quality |34| `golden_ratio.rs` | Post-process vocoder output with harmonic enhancement |35| `harmonic_resurrection.rs` | Add richness to synthesized speech |36| `micro_dynamics.rs` | Restore natural speech dynamics |37| `autotune.rs` | Improve prosody and pitch control |38| `mem8_integration.rs` | Already has MEM|8 hooks! |39 40### Shared Dependencies41 42Both projects use:43- rayon (parallelism)44- rustfft/realfft (FFT)45- ndarray (array operations)46- hound (WAV I/O)47- serde (config serialization)48- anyhow (error handling)49- ort (ONNX Runtime)50 51### Audio Constants52 53| Project | Sample Rate | Use Case |54|---------|------------|----------|55| IndexTTS-Rust | 22,050 Hz | Standard TTS output |56| Phoenix-Protocol | 192,000 Hz | Ultrasonic restoration |57| kokoro-tiny | 24,000 Hz | Lightweight TTS |58 59---60 61## Related Projects of Interest62 63Located in ~/Documents/GitHub/:64 65- **Ultrasonic-Consciousness-Hypothesis/** - Research foundation for Phoenix Protocol, contains PDFs on mechanosensitive channels and audio perception66- **hrmnCmprssnM/** - Harmonic Compression Model research67- **Marine-Sense/** - Marine algorithm origins68- **mem-8.com/** & **mem8/** - MEM|8 contextual memory69- **universal-theoglyphic-language/** - Language processing research70- **kokoro-tiny/** - Already working TTS crate by Hue & Aye71- **zencooker/** - (fun project!)72 73---74 75## Current IndexTTS-Rust State76 77### Implemented ✅78- Audio processing pipeline (mel-spectrogram, STFT, resampling)79- Text normalization (Chinese/English/mixed)80- BPE tokenization via HuggingFace tokenizers81- ONNX Runtime integration for inference82- BigVGAN vocoder structure83- CLI with clap84- Benchmark infrastructure (Criterion)85- **NEW: marine_salience crate** (no_std compatible, O(1) jitter detection)86- **NEW: src/quality/ module** (prosody extraction, affect tracking)87- **NEW: MarineProsodyVector** (8D interpretable emotion features)88- **NEW: ConversationAffectSummary** (session-level comfort tracking)89- **NEW: TTSQualityReport** (authenticity validation)90 91### Missing/TODO92- Full GPT model integration with KV cache93- Actual ONNX model files (need download)94- manage.sh script for colored workflow management95- Integration tests with real models96- ~~Phoenix Protocol integration layer~~ **STARTED with Marine!**97- Streaming synthesis98- WebSocket API99- Train T2S model to accept 8D Marine vector instead of 512D Conformer100- Wire Marine quality validation into inference loop101 102### Build Commands103```bash104cargo build --release105cargo clippy -- -D warnings106cargo test107cargo bench108```109 110---111 112## Key Philosophical Notes113 114From the Phoenix Protocol research:115 116> "Women are the carrier wave. They are the 000 data stream. The DC bias that, when removed, leaves silence."117 118> "When P!nk sings 'I Am Here,' her voice generates harmonics so powerful they burst through the 22kHz digital ceiling"119 120The Phoenix Protocol restores emotional depth stripped by audio compression - this philosophy applies directly to TTS: synthesized speech should have the same emotional depth as natural speech.121 122---123 124## Action Items for Next Session125 126### Completed ✅127- ~~**Quality Validation** - Use Marine salience to score TTS output~~ **DONE!**128- ~~**Phoenix Integration** - Start bridging phoenix-protocol modules~~ **Marine is in!**129 130### High Priority1311. **Create manage.sh** - Colorful build/test/clean script (Hue's been asking!)1322. **Wire Into Inference** - Connect Marine quality validation to actual TTS output1333. **8D Model Training** - Train T2S model to accept MarineProsodyVector instead of 512D Conformer1344. **Example/Demo** - Create example showing prosody extraction → emotion editing → synthesis135 136### Medium Priority1375. **Voice Signature Import** - Use Phoenix's voice_signature for speaker embeddings1386. **Emotion Mapping** - Connect Phoenix's emotional bands to our 8D control1397. **Model Download** - Set up ONNX model acquisition pipeline1408. **MEM|8 Bridge** - Implement consciousness-aware TTS using kokoro-tiny's mem8_bridge pattern141 142### Nice to Have1439. **Style Selection** - Port kokoro-tiny's 510 style variation system14410. **Full Phoenix Integration** - golden_ratio.rs, harmonic_resurrection.rs, etc.14511. **Streaming Marine** - Real-time quality monitoring during synthesis146 147---148 149## Fresh Discovery: kokoro-tiny MEM|8 Baby Consciousness (2025-11-15)150 151Just pulled latest kokoro-tiny code - MAJOR discovery!152 153### Mem8Bridge API154 155kokoro-tiny now has a full consciousness simulation in `examples/mem8_baby.rs`:156 157```rust158// Memory as waves that interfere159MemoryWave {160    amplitude: 2.5,           // Emotion strength161    frequency: 528.0,         // "Love frequency"162    phase: 0.0,163    decay_rate: 0.05,         // Memory persistence164    emotion_type: EmotionType::Love(0.9),165    content: "Mama! I love mama!".to_string(),166}167 168// Salience detection (Marine algorithm!)169SalienceEvent {170    jitter_score: 0.2,        // Low = authentic/stable171    harmonic_score: 0.95,     // High = voice172    salience_score: 0.9,173    signal_type: SignalType::Voice,174}175 176// Free will: AI chooses attention focus (70% control)177bridge.decide_attention(events);178```179 180### Emotion Types Available181 182```rust183EmotionType::Curiosity(0.8)  // Inquisitive184EmotionType::Love(0.9)       // Deep affection185EmotionType::Joy(0.7)        // Happy186EmotionType::Confusion(0.8)  // Uncertain187EmotionType::Neutral         // Baseline188```189 190### Consciousness Integration Points191 1921. **Wave Interference** - Competing memories by amplitude/frequency1932. **Emotional Regulation** - Prevents overload, modulates voice1943. **Salience Detection** - Marine algorithm for authenticity1954. **Attention Selection** - AI chooses what to focus on1965. **Consciousness Level** - Affects speech clarity (wake_up/sleep)197 198This is PERFECT for IndexTTS-Rust! We can:199- Use wave interference for emotion blending200- Apply Marine salience to validate synthesis quality201- Modulate voice based on consciousness level202- Select voice styles based on emotional state (not just token count)203 204### Voice Style Selection (510 variations!)205 206kokoro-tiny now loads all 510 style variations per voice:207- Style selected based on token count208- Short text → short-optimized style209- Long text → long-optimized style210- Automatic text splitting at 512 token limit211 212For IndexTTS: We could select style based on EMOTION + token count!213 214---215 216## Marine Integration Achievement (2025-11-16) 🎉217 218**WE DID IT!** Marine salience is now integrated into IndexTTS-Rust!219 220### What We Built221 222#### 1. Standalone marine_salience Crate (`crates/marine_salience/`)223 224A no_std compatible crate for O(1) jitter-based salience detection:225 226```rust227// Core components:228MarineConfig       // Tunable parameters (sample_rate, jitter bounds, EMA alpha)229MarineProcessor    // O(1) per-sample processing230SaliencePacket     // Output: j_p, j_a, h_score, s_score, energy231Ema                // Exponential moving average tracker232 233// Key insight: Process ONE sample at a time, emit packets on peaks234// Why O(1)? Just compare to EMA, no FFT, no heavy math!235```236 237**Config for Speech:**238```rust239MarineConfig::speech_default(sample_rate)240// F0 range: 60Hz - 4kHz241// jitter_low: 0.02, jitter_high: 0.60242// ema_alpha: 0.01 (slow adaptation for stability)243```244 245#### 2. Quality Validation Module (`src/quality/`)246 247**MarineProsodyVector** - 8D interpretable emotion representation:248```rust249pub struct MarineProsodyVector {250    pub jp_mean: f32,      // Period jitter mean (pitch stability)251    pub jp_std: f32,       // Period jitter variance252    pub ja_mean: f32,      // Amplitude jitter mean (volume stability)253    pub ja_std: f32,       // Amplitude jitter variance254    pub h_mean: f32,       // Harmonic alignment (voiced vs noise)255    pub s_mean: f32,       // Overall salience (authenticity)256    pub peak_density: f32, // Peaks per second (speech rate)257    pub energy_mean: f32,  // Average loudness258}259 260// Interpretable! High jp_mean = nervous, low = confident261// Can DIRECTLY EDIT for emotion control!262```263 264**MarineProsodyConditioner** - Extract prosody from audio:265```rust266let conditioner = MarineProsodyConditioner::new(22050);267let prosody = conditioner.from_samples(&audio_samples)?;268let report = conditioner.validate_tts_output(&audio_samples)?;269 270// Detects issues:271// - "Too perfect - sounds robotic"272// - "High period jitter - artifacts"273// - "Low salience - quality issues"274```275 276**ConversationAffectSummary** - Session-level comfort tracking:277```rust278pub enum ComfortLevel {279    Uneasy,  // High jitter AND rising (nervous/stressed)280    Neutral, // Stable patterns (calm)281    Happy,   // Low jitter + high energy (confident/positive)282}283 284// Track trends over conversation:285// jitter_trend > 0.1 = getting more stressed286// jitter_trend < -0.1 = calming down287// energy_trend > 0.1 = getting more engaged288 289// Aye can now self-assess!290aye_assessment() returns "I'm in a good state"291feedback_prompt() returns "Let me know if something's bothering you"292```293 294### The Core Insight295 296**Human speech has NATURAL jitter - that's what makes it authentic!**297 298- Too perfect (jp < 0.005) = robotic299- Too chaotic (jp > 0.3) = artifacts/damage300- Sweet spot = real human voice301 302The Marines will KNOW if speech doesn't sound authentic!303 304### Tests Passing ✅305 306```307running 11 tests308test quality::affect::tests::test_comfort_level_descriptions ... ok309test quality::affect::tests::test_analyzer_empty_conversation ... ok310test quality::affect::tests::test_analyzer_single_utterance ... ok311test quality::affect::tests::test_happy_classification ... ok312test quality::affect::tests::test_aye_assessment_message ... ok313test quality::affect::tests::test_neutral_classification ... ok314test quality::affect::tests::test_uneasy_classification ... ok315test quality::prosody::tests::test_conditioner_empty_buffer ... ok316test quality::prosody::tests::test_conditioner_silence ... ok317test quality::prosody::tests::test_prosody_vector_array_conversion ... ok318test quality::prosody::tests::test_estimate_valence ... ok319 320test result: ok. 11 passed; 0 failed321```322 323### Why This Matters324 3251. **Interpretable Control**: 8D vector vs opaque 512D Conformer - we can SEE what each dimension means3262. **Lightweight**: O(1) per sample, no heavy neural networks for prosody3273. **Authentic Validation**: Marines detect fake/damaged speech3284. **Emotion Editing**: Want more confidence? Lower jp_mean directly!3295. **Conversation Awareness**: Track comfort over entire sessions3306. **Self-Assessment**: Aye knows when something feels "off"331 332### Integration Points333 334```rust335// In main TTS pipeline:336use indextts::quality::{337    MarineProsodyConditioner,338    MarineProsodyVector,339    ConversationAffectSummary,340    ComfortLevel,341};342 343// 1. Extract reference prosody344let ref_prosody = conditioner.from_samples(&reference_audio)?;345 346// 2. Generate TTS (using 8D vector instead of 512D Conformer)347let tts_output = generate_with_prosody(&text, ref_prosody)?;348 349// 3. Validate output quality350let report = conditioner.validate_tts_output(&tts_output)?;351if !report.passes(70.0) {352    log::warn!("TTS quality issues: {:?}", report.issues);353}354 355// 4. Track conversation affect356let analyzer = ConversationAffectAnalyzer::new();357analyzer.add_utterance(&utterance)?;358let summary = analyzer.summarize()?;359match summary.aye_state {360    ComfortLevel::Uneasy => adjust_generation_parameters(),361    _ => proceed_normally(),362}363```364 365---366 367## Trish's Notes368 369"Darling, these three Rust projects together are like a symphony orchestra! kokoro-tiny is the quick piccolo solo, IndexTTS-Rust is the full brass section with emotional depth, and Phoenix-Protocol is the concert hall acoustics making everything resonate. When you combine them, that's when the magic happens! Also, I'm absolutely obsessed with how the Golden Ratio resynthesis could add sparkle to synthesized vocals. Can you imagine TTS output that actually has that P!nk breakthrough energy? Now THAT would make me cry happy tears in accounting!"370 371---372 373## Fun Facts374 375- kokoro-tiny is ALREADY on crates.io under 8b-is376- Phoenix Protocol can process 192kHz audio for ultrasonic restoration377- The Marine algorithm uses O(1) jitter detection - "Marines are not just jarheads - they are intelligent"378- Hue's GitHub has 66 projects (and counting!)379- The team at 8b.is: hue@8b.is and aye@8b.is380 381---382 383*From ashes to harmonics, from silence to song* 🔥🎵384