ThreadAbort/IndexTTS-Rust
77
1╔════════════════════════════════════════════════════════════════════════════════╗2║ DETAILED SOURCE FILE LISTING BY CATEGORY ║3╚════════════════════════════════════════════════════════════════════════════════╝4 5MAIN INFERENCE PIPELINE FILES6═════════════════════════════════════════════════════════════════════════════════7 8/home/user/IndexTTS-Rust/indextts/infer_v2.py (739 LINES) ⭐⭐⭐ CRITICAL9├─ Purpose: Main TTS inference class (IndexTTS2)10├─ Key Classes:11│ ├─ QwenEmotion (emotion text-to-vector conversion)12│ ├─ IndexTTS2 (main inference class)13│ └─ Helper functions for emotion/audio processing14├─ Key Methods:15│ ├─ __init__() - Initialize all models and codecs16│ ├─ infer() - Single text generation with emotion control17│ ├─ infer_fast() - Parallel segment generation18│ ├─ get_emb() - Extract semantic embeddings19│ ├─ remove_long_silence() - Silence token removal20│ ├─ insert_interval_silence() - Silence insertion21│ └─ Cache management for repeated generation22├─ Models Loaded:23│ ├─ UnifiedVoice (GPT model for mel token generation)24│ ├─ W2V-BERT (semantic feature extraction)25│ ├─ RepCodec (semantic codec)26│ ├─ S2Mel model (semantic-to-mel conversion)27│ ├─ CAMPPlus (speaker embedding)28│ ├─ BigVGAN vocoder29│ ├─ Qwen-based emotion model30│ └─ Emotion/speaker matrices31└─ External Dependencies: torch, transformers, librosa, safetensors32 33/home/user/IndexTTS-Rust/webui.py (18KB) ⭐⭐⭐ WEB INTERFACE34├─ Purpose: Gradio-based web UI for IndexTTS35├─ Key Components:36│ ├─ Model initialization (IndexTTS2 instance)37│ ├─ Language selection (Chinese/English)38│ ├─ Emotion control modes (4 modes)39│ ├─ Example case loading from cases.jsonl40│ ├─ Progress bar integration41│ └─ Output management42├─ Features:43│ ├─ Real-time inference44│ ├─ Multiple emotion control methods45│ ├─ Batch processing46│ ├─ Task caching47│ ├─ i18n support48│ └─ Pre-loaded example cases49└─ Web Framework: Gradio 5.34.150 51/home/user/IndexTTS-Rust/indextts/cli.py (64 LINES)52├─ Purpose: Command-line interface53├─ Usage: python -m indextts.cli <text> -v <voice.wav> -o <output.wav> [options]54├─ Arguments:55│ ├─ text: Text to synthesize56│ ├─ -v/--voice: Voice reference audio57│ ├─ -o/--output_path: Output file path58│ ├─ -c/--config: Config file path59│ ├─ --model_dir: Model directory60│ ├─ --fp16: Use FP16 precision61│ ├─ -d/--device: Device (cpu/cuda/mps/xpu)62│ └─ -f/--force: Force overwrite63└─ Uses: IndexTTS (v1 model)64 65TEXT PROCESSING & NORMALIZATION FILES66═════════════════════════════════════════════════════════════════════════════════67 68/home/user/IndexTTS-Rust/indextts/utils/front.py (700 LINES) ⭐⭐⭐ CRITICAL69├─ Purpose: Text normalization and tokenization70├─ Key Classes:71│ ├─ TextNormalizer (700+ lines)72│ │ ├─ Pattern Definitions:73│ │ │ ├─ PINYIN_TONE_PATTERN (regex for pinyin with tones 1-5)74│ │ │ ├─ NAME_PATTERN (regex for Chinese names)75│ │ │ └─ ENGLISH_CONTRACTION_PATTERN (regex for 's contractions)76│ │ ├─ Methods:77│ │ │ ├─ normalize() - Main normalization78│ │ │ ├─ use_chinese() - Language detection79│ │ │ ├─ save_pinyin_tones() - Extract pinyin with tones80│ │ │ ├─ restore_pinyin_tones() - Restore pinyin81│ │ │ ├─ save_names() - Extract names82│ │ │ ├─ restore_names() - Restore names83│ │ │ ├─ correct_pinyin() - Phoneme correction (jqx→v)84│ │ │ └─ char_rep_map - Character replacement dictionary85│ │ └─ Normalizers:86│ │ ├─ zh_normalizer (Chinese) - Uses WeTextProcessing/wetext87│ │ └─ en_normalizer (English) - Uses tn library88│ │89│ └─ TextTokenizer (200+ lines)90│ ├─ Methods:91│ │ ├─ encode() - Text to token IDs92│ │ ├─ decode() - Token IDs to text93│ │ ├─ convert_tokens_to_ids()94│ │ ├─ convert_ids_to_tokens()95│ │ └─ Vocab management96│ ├─ Special Tokens:97│ │ ├─ BOS: "<s>" (ID 0)98│ │ ├─ EOS: "</s>" (ID 1)99│ │ └─ UNK: "<unk>"100│ └─ Tokenizer: SentencePiece (BPE-based)101├─ Language Support:102│ ├─ Chinese (simplified & traditional)103│ ├─ English104│ └─ Mixed Chinese-English105└─ Critical Pattern Matching:106 ├─ Pinyin tone detection107 ├─ Name entity detection108 ├─ Email matching109 ├─ Character replacement110 └─ Punctuation handling111 112GPT MODEL ARCHITECTURE FILES113═════════════════════════════════════════════════════════════════════════════════114 115/home/user/IndexTTS-Rust/indextts/gpt/model_v2.py (747 LINES) ⭐⭐⭐ CRITICAL116├─ Purpose: UnifiedVoice GPT-based TTS model117├─ Key Classes:118│ ├─ UnifiedVoice (700+ lines)119│ │ ├─ Architecture:120│ │ │ ├─ Input Embeddings: Text (256 vocab), Mel (8194 vocab)121│ │ │ ├─ Position Embeddings: Learned embeddings for mel/text122│ │ │ ├─ GPT Transformer: Configurable layers/heads123│ │ │ ├─ Conditioning Encoder: Conformer or Perceiver-based124│ │ │ ├─ Emotion Conditioning: Separate conformer + perceiver125│ │ │ └─ Output Heads: Text prediction, Mel prediction126│ │ │127│ │ ├─ Parameters:128│ │ │ ├─ layers: 8 (transformer depth)129│ │ │ ├─ model_dim: 512 (embedding dimension)130│ │ │ ├─ heads: 8 (attention heads)131│ │ │ ├─ max_text_tokens: 120132│ │ │ ├─ max_mel_tokens: 250133│ │ │ ├─ number_mel_codes: 8194134│ │ │ ├─ condition_type: "conformer_perceiver" or "conformer_encoder"135│ │ │ └─ Various activation functions136│ │ │137│ │ ├─ Key Methods:138│ │ │ ├─ forward() - Forward pass139│ │ │ ├─ post_init_gpt2_config() - Initialize for inference140│ │ │ ├─ generate_mel() - Mel token generation141│ │ │ ├─ forward_with_cond_scale() - With classifier-free guidance142│ │ │ └─ Cache management143│ │ │144│ │ └─ Conditioning System:145│ │ ├─ Speaker conditioning via mel spectrogram146│ │ ├─ Conformer encoder for speaker features147│ │ ├─ Perceiver for attention pooling148│ │ ├─ Emotion conditioning (separate pathway)149│ │ └─ Emotion vector support (8-dimensional)150│ │151│ ├─ ResBlock (40+ lines)152│ │ ├─ Conv1d layers with GroupNorm153│ │ └─ ReLU activation with residual connection154│ │155│ ├─ GPT2InferenceModel (200+ lines)156│ │ ├─ Inference wrapper for GPT2157│ │ ├─ KV cache support158│ │ ├─ Model parallelism support159│ │ └─ Token-by-token generation160│ │161│ ├─ ConditioningEncoder (30 lines)162│ │ ├─ Conv1d initialization163│ │ ├─ Attention blocks164│ │ └─ Optional mean pooling165│ │166│ ├─ MelEncoder (30 lines)167│ │ ├─ Conv1d layers168│ │ ├─ ResBlocks169│ │ └─ 4x reduction170│ │171│ ├─ LearnedPositionEmbeddings (15 lines)172│ │ └─ Learnable positional embeddings173│ │174│ └─ build_hf_gpt_transformer() (20 lines)175│ └─ Builds HuggingFace GPT2 with custom embeddings176│177├─ External Dependencies: torch, transformers, indextts.gpt modules178└─ Critical Inference Parameters:179 ├─ Temperature control for generation180 ├─ Top-k/top-p sampling181 ├─ Classifier-free guidance scale182 └─ Generation length limits183 184/home/user/IndexTTS-Rust/indextts/gpt/conformer_encoder.py (520 LINES) ⭐⭐185├─ Purpose: Conformer-based speaker conditioning encoder186├─ Key Classes:187│ ├─ ConformerEncoder (main)188│ │ ├─ Modules:189│ │ │ ├─ Subsampling layer (Conv2d)190│ │ │ ├─ Positional encoding191│ │ │ ├─ Conformer blocks192│ │ │ ├─ Layer normalization193│ │ │ └─ Optional projection layer194│ │ │195│ │ ├─ Configuration Parameters:196│ │ │ ├─ input_size: 1024 (mel spectrogram bins)197│ │ │ ├─ output_size: depends on config198│ │ │ ├─ linear_units: hidden dim for FFN199│ │ │ ├─ attention_heads: 8200│ │ │ ├─ num_blocks: 4201│ │ │ └─ input_layer: "linear" or "conv2d"202│ │ │203│ │ └─ Architecture: Conv → Pos Enc → [Conformer Block] * N → LayerNorm204│ │205│ ├─ ConformerBlock (80+ lines)206│ │ ├─ Residual connections207│ │ ├─ FFN → Attention → Conv → FFN structure208│ │ ├─ Feed-forward network (2-layer with dropout)209│ │ ├─ Multi-head self-attention210│ │ ├─ Convolution module (depthwise)211│ │ └─ Layer normalization212│ │213│ ├─ ConvolutionModule (50 lines)214│ │ ├─ Pointwise Conv 1x1215│ │ ├─ Depthwise Conv with kernel_size (e.g., 15)216│ │ ├─ Batch normalization or layer normalization217│ │ ├─ Activation (ReLU/SiLU)218│ │ └─ Projection219│ │220│ ├─ PositionwiseFeedForward (15 lines)221│ │ ├─ Dense layer (idim → hidden)222│ │ ├─ Activation (ReLU)223│ │ ├─ Dropout224│ │ └─ Dense layer (hidden → idim)225│ │226│ └─ MultiHeadedAttention (custom)227│ ├─ Scaled dot-product attention228│ ├─ Multiple heads229│ └─ Optional relative position bias230│231├─ External Dependencies: torch, custom conformer modules232└─ Use Case: Processing mel spectrogram to extract speaker features233 234/home/user/IndexTTS-Rust/indextts/gpt/perceiver.py (317 LINES) ⭐⭐235├─ Purpose: Perceiver resampler for attention pooling236├─ Key Classes:237│ ├─ PerceiverResampler (250+ lines)238│ │ ├─ Architecture:239│ │ │ ├─ Learnable latent queries240│ │ │ ├─ Cross-attention layers241│ │ │ ├─ Feed-forward networks242│ │ │ └─ Layer normalization243│ │ │244│ │ ├─ Parameters:245│ │ │ ├─ dim: 512 (embedding dimension)246│ │ │ ├─ dim_context: 512 (context dimension)247│ │ │ ├─ num_latents: 32 (number of latent queries)248│ │ │ ├─ num_latent_channels: 64249│ │ │ ├─ num_layers: 6250│ │ │ ├─ ff_mult: 4 (FFN expansion)251│ │ │ └─ heads: 8252│ │ │253│ │ ├─ Key Methods:254│ │ │ ├─ forward() - Attend and pool255│ │ │ └─ _cross_attend_block() - Single cross-attention layer256│ │ │257│ │ └─ Cross-Attention Mechanism:258│ │ ├─ Queries: Learnable latents259│ │ ├─ Keys/Values: Input context260│ │ ├─ Output: Pooled features (num_latents × dim)261│ │ └─ FFN projection for dimension mixing262│ │263│ └─ FeedForward (15 lines)264│ ├─ Dense (dim → hidden)265│ ├─ GELU activation266│ └─ Dense (hidden → dim)267│268├─ External Dependencies: torch, einsum operations269└─ Use Case: Pool conditioning encoder output to fixed-size representation270 271VOCODER & AUDIO SYNTHESIS FILES272═════════════════════════════════════════════════════════════════════════════════273 274/home/user/IndexTTS-Rust/indextts/BigVGAN/models.py (1000+ LINES) ⭐⭐⭐275├─ Purpose: BigVGAN neural vocoder for mel-to-audio conversion276├─ Key Classes:277│ ├─ BigVGAN (400+ lines)278│ │ ├─ Architecture:279│ │ │ ├─ Initial Conv1d (80 mel bins → 192 channels)280│ │ │ ├─ Upsampling layers (transposed conv)281│ │ │ ├─ AMP blocks (anti-aliased multi-period)282│ │ │ ├─ Final Conv1d (channels → 1 waveform)283│ │ │ └─ Tanh activation for output284│ │ │285│ │ ├─ Upsampling: 4x → 8x → 8x → 4x (256x total)286│ │ │ ├─ Maps from 22050 Hz mel frames to audio samples287│ │ │ ├─ Kernel sizes: [16, 16, 4, 4]288│ │ │ └─ Padding: [6, 6, 2, 2]289│ │ │290│ │ ├─ Parameters:291│ │ │ ├─ num_mels: 80292│ │ │ ├─ num_freq: 513293│ │ │ ├─ num_mels: 80294│ │ │ ├─ n_fft: 1024295│ │ │ ├─ hop_size: 256296│ │ │ ├─ win_size: 1024297│ │ │ ├─ sampling_rate: 22050298│ │ │ ├─ freq_min: 0299│ │ │ ├─ freq_max: None300│ │ │ └─ use_cuda_kernel: bool301│ │ │302│ │ ├─ Key Methods:303│ │ │ ├─ forward() - Mel → audio waveform304│ │ │ ├─ from_pretrained() - Load from HuggingFace305│ │ │ ├─ remove_weight_norm() - Remove spectral normalization306│ │ │ └─ eval() - Set to evaluation mode307│ │ │308│ │ └─ Special Features:309│ │ ├─ Weight normalization for training stability310│ │ ├─ Spectral normalization option311│ │ ├─ CUDA kernel support for activation functions312│ │ ├─ Snake/SnakeBeta activation (periodic)313│ │ └─ Anti-aliasing filters for high-quality upsampling314│ │315│ ├─ AMPBlock1 (50 lines)316│ │ ├─ Architecture: Conv1d × 2 with activations317│ │ ├─ Multiple dilation patterns [1, 3, 5]318│ │ ├─ Residual connections319│ │ ├─ Activation1d wrapper for anti-aliasing320│ │ └─ Weight normalization321│ │322│ ├─ AMPBlock2 (40 lines)323│ │ ├─ Similar to AMPBlock1 but simpler324│ │ ├─ Dilation patterns [1, 3]325│ │ └─ Residual connections326│ │327│ ├─ Activation1d (custom, from alias_free_activation/)328│ │ ├─ Applies activation function (Snake/SnakeBeta)329│ │ ├─ Optional anti-aliasing filter330│ │ └─ Optional CUDA kernel for efficiency331│ │332│ ├─ Snake Activation (from activations.py)333│ │ ├─ Formula: x + (1/alpha) * sin²(alpha * x)334│ │ ├─ Periodic nonlinearity335│ │ └─ Learnable alpha parameter336│ │337│ └─ SnakeBeta Activation (from activations.py)338│ ├─ More complex periodic activation339│ └─ Improved harmonic modeling340│341├─ External Dependencies: torch, scipy, librosa342└─ Model Size: ~100 MB (pretrained weights)343 344/home/user/IndexTTS-Rust/indextts/s2mel/modules/audio.py (83 LINES)345├─ Purpose: Mel-spectrogram computation (DSP)346├─ Key Functions:347│ ├─ load_wav() - Load WAV file with scipy348│ ├─ mel_spectrogram() - Compute mel spectrogram349│ │ ├─ Parameters:350│ │ │ ├─ y: waveform tensor351│ │ │ ├─ n_fft: 1024352│ │ │ ├─ num_mels: 80353│ │ │ ├─ sampling_rate: 22050354│ │ │ ├─ hop_size: 256355│ │ │ ├─ win_size: 1024356│ │ │ ├─ fmin: 0357│ │ │ └─ fmax: None or 8000358│ │ │359│ │ ├─ Process:360│ │ │ 1. Pad input with reflect padding361│ │ │ 2. Compute STFT (Short-Time Fourier Transform)362│ │ │ 3. Convert to magnitude spectrogram363│ │ │ 4. Apply mel filterbank (librosa)364│ │ │ 5. Apply dynamic range compression (log)365│ │ │ └─ Output: [1, 80, T] tensor366│ │ │367│ │ └─ Caching:368│ │ ├─ Caches mel filterbank matrices369│ │ ├─ Caches Hann windows370│ │ └─ Device-specific caching371│ │372│ ├─ dynamic_range_compression() - Log compression373│ ├─ dynamic_range_decompression() - Inverse374│ └─ spectral_normalize/denormalize()375│376├─ Critical DSP Parameters:377│ ├─ STFT Window: Hann window378│ ├─ FFT Size: 1024379│ ├─ Hop Size: 256 (11.6 ms at 22050 Hz)380│ ├─ Mel Bins: 80 (perceptual scale)381│ ├─ Min Freq: 0 Hz382│ └─ Max Freq: Variable (8000 Hz or Nyquist)383│384└─ External Dependencies: torch, librosa, scipy385 386SEMANTIC CODEC & FEATURE EXTRACTION FILES387═════════════════════════════════════════════════════════════════════════════════388 389/home/user/IndexTTS-Rust/indextts/utils/maskgct_utils.py (250 LINES)390├─ Purpose: Build and manage semantic codecs391├─ Key Functions:392│ ├─ build_semantic_model()393│ │ ├─ Loads: facebook/w2v-bert-2.0 model394│ │ ├─ Extracts: wav2vec 2.0 BERT embeddings395│ │ ├─ Returns: model, mean, std (for normalization)396│ │ └─ Output: 1024-dimensional embeddings397│ │398│ ├─ build_semantic_codec()399│ │ ├─ Creates: RepCodec (residual vector quantization)400│ │ ├─ Quantizes: Semantic embeddings401│ │ ├─ Returns: Codec model402│ │ └─ Output: Discrete tokens403│ │404│ ├─ build_s2a_model()405│ │ ├─ Builds: MaskGCT_S2A (semantic-to-acoustic)406│ │ └─ Maps: Semantic codes → acoustic codes407│ │408│ ├─ build_acoustic_codec()409│ │ ├─ Encoder: Encodes acoustic features410│ │ ├─ Decoder: Decodes codes → audio411│ │ └─ Multiple codec variants412│ │413│ └─ Inference_Pipeline (class)414│ ├─ Combines all codecs415│ ├─ Methods:416│ │ ├─ get_emb() - Get semantic embeddings417│ │ ├─ get_scode() - Quantize to semantic codes418│ │ ├─ semantic2acoustic() - Convert codes419│ │ └─ s2a_inference() - Full pipeline420│ └─ Diffusion-based generation options421│422├─ External Dependencies: torch, transformers, huggingface_hub423└─ Pre-trained Models:424 ├─ W2V-BERT-2.0: 614M parameters425 ├─ MaskGCT: From amphion/MaskGCT426 └─ Various codec checkpoints427 428CONFIGURATION & UTILITY FILES429═════════════════════════════════════════════════════════════════════════════════430 431/home/user/IndexTTS-Rust/indextts/utils/checkpoint.py (50 LINES)432├─ Purpose: Load model checkpoints433├─ Key Functions:434│ ├─ load_checkpoint() - Load weights into model435│ └─ Device handling (CPU/GPU/XPU/MPS)436└─ Supported Formats: .pth, .safetensors437 438/home/user/IndexTTS-Rust/indextts/utils/arch_util.py439├─ Purpose: Architecture utility modules440├─ Key Classes:441│ └─ AttentionBlock - Generic attention layer442└─ Used in: Conditioning encoder, other modules443 444/home/user/IndexTTS-Rust/indextts/utils/xtransformers.py (1,600 LINES)445├─ Purpose: Extended transformer utilities446├─ Key Components:447│ ├─ Advanced attention mechanisms448│ ├─ Relative position bias449│ ├─ Cross-attention patterns450│ └─ Various position encoding schemes451└─ Used in: GPT model, encoders452 453TESTING FILES454═════════════════════════════════════════════════════════════════════════════════455 456/home/user/IndexTTS-Rust/tests/regression_test.py457├─ Test Cases:458│ ├─ Chinese text with pinyin tones (晕 XUAN4)459│ ├─ English text460│ ├─ Mixed Chinese-English461│ ├─ Long-form text with multiple sentences462│ ├─ Named entities (Joseph Gordon-Levitt)463│ ├─ Chinese names (约瑟夫·高登-莱维特)464│ └─ Extended passages for robustness465├─ Inference Modes:466│ ├─ Single inference (infer)467│ └─ Fast inference (infer_fast)468└─ Output: WAV files in outputs/ directory469 470/home/user/IndexTTS-Rust/tests/padding_test.py471├─ Test Scenarios:472│ ├─ Variable length inputs473│ ├─ Batch processing474│ ├─ Edge cases475│ └─ Padding handling476└─ Purpose: Ensure robust padding mechanics477 478═════════════════════════════════════════════════════════════════════════════════479 480KEY ALGORITHMS SUMMARY:481 4821. TEXT PROCESSING:483 - Regex-based pattern matching for pinyin/names484 - Character-level CJK tokenization485 - SentencePiece BPE encoding486 - Language detection (Chinese vs English)487 4882. FEATURE EXTRACTION:489 - W2V-BERT semantic embeddings (1024-dim)490 - RepCodec quantization491 - Mel-spectrogram (STFT-based, 80-dim)492 - CAMPPlus speaker embeddings (192-dim)493 4943. SEQUENCE GENERATION:495 - GPT-based autoregressive generation496 - Conformer speaker conditioning497 - Perceiver pooling for attention498 - Classifier-free guidance (optional)499 - Temperature/top-k/top-p sampling500 5014. AUDIO SYNTHESIS:502 - Transposed convolution upsampling (256x)503 - Anti-aliased activation functions504 - Residual connections505 - Weight/spectral normalization506 5075. EMOTION CONTROL:508 - 8-dimensional emotion vectors509 - Text-based emotion detection (via Qwen)510 - Audio-based emotion extraction511 - Emotion matrix interpolation512 513═════════════════════════════════════════════════════════════════════════════════514