BlueWaveSemi45/DramaboxCPU

1---2title: DramaBox3emoji: 🎭4colorFrom: red5colorTo: indigo6sdk: gradio7sdk_version: 6.14.08app_file: app.py9pinned: true10license: other11license_name: ltx-2-community12license_link: https://huggingface.co/ResembleAI/Dramabox/blob/main/LICENSE13hf_oauth: false14short_description: Expressive TTS with voice cloning — DramaBox demo15---16 17# DramaBox — Expressive TTS with Voice Cloning18 19> **Built on [LTX-2](https://github.com/Lightricks/LTX-2) by Lightricks.**20> DramaBox is **Resemble AI's** expressive TTS, trained on top of the LTX-2.3 audio branch under the LTX-2 Community License. Huge thanks to the Lightricks team for open-sourcing the base.21 22Prompt-driven TTS with voice cloning. The prompt itself controls speaker identity, emotion, delivery style, laughs, sighs, pauses and transitions; an optional 10-second voice reference clones the target timbre. DramaBox is an IC-LoRA fine-tune of the **LTX-2.3 3.3B audio-only** model.23 24| | |25|---|---|26| 🤗 **Model** | [`ResembleAI/Dramabox`](https://huggingface.co/ResembleAI/Dramabox) |27| 🎭 **Demo Space** | [`ResembleAI/Dramabox`](https://huggingface.co/spaces/ResembleAI/Dramabox) (ZeroGPU) |28| 🏗️ **Base model** | [`Lightricks/LTX-2`](https://huggingface.co/Lightricks/LTX-2) |29| 📜 **License** | LTX-2 Community License — see [`LICENSE`](LICENSE) |30 31## Models32 33Auto-downloaded from the HF model repo on first run.34 35| File | Size | Description |36|---|---|---|37| `dramabox-dit-v1.safetensors` | 6.6 GB | DiT transformer (LoRA already merged into base) |38| `dramabox-audio-components.safetensors` | 1.9 GB | Audio embeddings connector + audio text projection + audio VAE + vocoder |39| [`unsloth/gemma-3-12b-it-bnb-4bit`](https://huggingface.co/unsloth/gemma-3-12b-it-bnb-4bit) | ~8 GB | Text encoder |40 41**VRAM**: ~24 GB peak · **Speed**: ~2.5 s / generation (warm server, H100)42 43## Quick Start44 45### Warm server (recommended)46 47```python48from src.inference_server import TTSServer49 50server = TTSServer(device="cuda")51 52server.generate_to_file(53    prompt='A woman speaks warmly, "Hello, how are you today?" She laughs, "Hahaha, it is so good to see you!"',54    output="output.wav",55    voice_ref="reference.wav",   # optional, 10+ seconds56)57```58 59### CLI60 61```bash62python src/inference.py \63  --voice-sample reference.wav \64  --prompt 'A woman speaks warmly, "Hello, how are you today?"' \65  --output output.wav \66  --cfg-scale 2.5 --stg-scale 1.567```68 69### Gradio app70 71```bash72CUDA_VISIBLE_DEVICES=4 python app.py73```74 75## Inference Settings76 77| Parameter | Default | Notes |78|---|---|---|79| `cfg-scale` | 2.5 | Lower = more natural, higher = more text-faithful |80| `stg-scale` | 1.5 | Skip-token guidance |81| `rescale` | 0 | No rescaling |82| `modality` | 1 | No modality guidance |83| `duration-multiplier` | 1.1 | 10% breathing room on auto-estimated length |84| `steps` | 30 | Euler flow matching |85 86## Prompt Writing Guide87 88**Structure:** `<speaker description>, "<dialogue>" <action direction> "<more dialogue>"`89 90**Inside quotes** (model produces actual sounds):91- Laughs: `"Hahaha"` `"Hehehe"` (always one word, never separated)92- Sounds: `"Mmmmm"` `"Ugh"` `"Argh"` `"Ahhh"` `"Hmm"`93 94**Outside quotes** (stage directions):95- `She sighs deeply.` · `He gulps nervously.` · `A long pause.`96- `Her voice cracks.` · `He clears his throat.` · `She scoffs.`97 98**Avoid inside quotes** (model speaks them literally): `Ahem`, `Pfft`, `Sigh`, `Gasp`, `Cough`.99 100**Tips**101- Match gender/age in the speaker description to the voice reference102- Break long dialogue into segments with action directions in between103- End the prompt at the last closing quote mark (no trailing description)104 105## Watermarking106 107Every audio output from `inference.py` and `inference_server.TTSServer.generate_to_file` is automatically watermarked with [Resemble Perth](https://github.com/resemble-ai/Perth) — an imperceptible neural watermark that survives MP3 compression, audio editing, and common manipulations while maintaining nearly 100% detection accuracy.108 109```python110import perth, librosa111wav, sr = librosa.load("output.wav", sr=None, mono=True)112detector = perth.PerthImplicitWatermarker()113print(detector.get_watermark(wav, sample_rate=sr))   # confidence ≈ 1.0114```115 116Pass `--no-watermark` to `inference.py` (or `watermark=False` to `generate_to_file`) to disable for debugging.117 118## Training a LoRA on top of DramaBox119 120You can fine-tune your own LoRA using DramaBox itself as the base — no need to start from raw LTX-2.3. Useful for adding a specific speaker, language flavour, or style on top of the existing expressive prior.121 122### 1. Prepare your index file123 124The preprocessor accepts four formats. The `text` field is the **target transcript**; if you want to attach a scene-style prompt (the part the model conditions on at inference time), prepend it to the transcript in the same format the model was trained on:125 126> `A woman speaks warmly, "<your transcript here>"`127 128Both forms are supported — with or without the prompt wrapper. Without the wrapper the model treats the entry as plain text-to-speech.129 130**Format A — `manifest` (JSONL)** — recommended for new datasets:131 132```jsonl133{"audio_filepath": "wavs/spk01_001.wav", "text": "A woman speaks warmly, \"Hello, how are you today?\""}134{"audio_filepath": "wavs/spk01_002.wav", "text": "Hello, how are you today?"}135{"audio_filepath": "wavs/spk02_001.flac", "text": "An exhausted father sighs, \"Sweetie, daddy is asking very nicely.\"", "duration": 4.7}136```137 138Fields: `audio_filepath` (or `audio_path`) is required, `text` (or `transcript`) is required, `duration` is optional.139 140**Format B — `tsv`** — simplest, one line per sample:141 142```143wavs/spk01_001.wav	A woman speaks warmly, "Hello, how are you today?"144wavs/spk01_002.wav	Hello, how are you today?145```146 147**Format C — `gemini_synthetic`** — `~`-separated, used for prompted synthetic data:148 149```150id~speaker~lang~sr~samples~dur~phonemes~text151spk01_001~spk01~en~24000~93000~3.875~_~A woman speaks warmly, "Hello, how are you today?"152```153 154**Format D — `libriheavy`** — `~`-separated, for unprompted text-only data:155 156```157id~speaker~lang~samples~dur_ms~phonemes~text158spk01_001~spk01~en~93000~3875~_~Hello, how are you today?159```160 161### 2. Preprocess162 163```bash164python src/preprocess.py \165  --dataset-type manifest \166  --index your_data.jsonl \167  --audio-dir /path/to/wavs \168  --output-dir /path/to/preprocessed/ \169  --checkpoint /path/to/dramabox-audio-components.safetensors \170  --gemma-root /path/to/gemma-3-12b-it-bnb-4bit/ \171  --max-duration 20.0 --min-duration 2.0172```173 174Output layout (training-ready `.pt` files):175 176```177preprocessed/178├── audio_latents/sample_*.pt     # Audio VAE-encoded latents179├── conditions/sample_*.pt        # Gemma text embeddings180└── latents/sample_*.pt           # Dummy video latents (placeholder)181```182 183### 3. Train184 185Copy `configs/training_args.example.yaml`, point `data_dir` / `speaker_index` at your preprocessed output, set `checkpoint` + `full_checkpoint` to the DramaBox files, then launch with HuggingFace `accelerate`. Any flag passed on the CLI overrides the YAML.186 187```bash188accelerate launch src/train.py \189  --config configs/training_args.example.yaml190```191 192The trainer attaches a fresh LoRA to the audio branch on top of the DramaBox checkpoint. LoRA targets: `audio_attn1.{to_q,to_k,to_v,to_out.0}` + `audio_ff.{net.0.proj,net.2}` × 48 transformer blocks (288 LoRA pairs total). Default rank 128 / alpha 128 / dropout 0.1, cosine LR schedule from 1e-4 with 500-step warmup over 10k steps.193 194To monitor training, set `val_config: configs/val_config.example.yaml` in your training YAML — `src/validate.py` is then spawned at every save step to generate one wav per speaker entry, so you can A/B listen during the run.195 196### Inference with your trained LoRA197 198```bash199python src/inference.py \200  --lora /path/to/your/lora_step_5000.safetensors \201  --voice-sample reference.wav \202  --prompt 'A woman speaks warmly, "..."' \203  --output output.wav204```205 206Always load the LoRA at inference rather than pre-merging it — pre-merged checkpoints have produced degraded output in our runs.207 208## Language209 210English.211 212## License & acknowledgement213 214DramaBox is a Resemble AI fine-tune of [LTX-2](https://github.com/Lightricks/LTX-2). Distributed under the LTX-2 Community License Agreement — see [`LICENSE`](LICENSE). Thanks again to Lightricks for releasing the base model.215