NandemoGHS/Anime-XCodec2

Anime‑XCodec2: Japanese Fine‑Tuned Variant of XCodec2


TL;DR: Anime‑XCodec2 is a fine‑tuned variant of HKUSTAudio/xcodec2, trained on \~25k hours of Japanese anime/game‑style voices.

Only the decoder was updated; the encoder and codebook remain frozen, so speech tokens are identical to the original XCodec2. This makes the model a drop‑in decoder for downstream systems that already work with XCodec2 tokens (e.g., Llasa).

🔗 Quick Links

• —Demo (Gradio / Hugging Face Spaces): https://huggingface.co/spaces/OmniAICreator/Anime-XCodec2-Demo
• —Baseline model (pretrained): HKUSTAudio/xcodec2
• —This repository (fine‑tuned): NandemoGHS/Anime‑XCodec2
• —Training Logs (Weights & Biases): View Report

1) Model Summary

• —What it is: A neural speech codec / speech tokenizer model based on XCodec2 with a decoder fine‑tuned for Japanese speech, particularly anime/game‑style voices.
• —Training scope: Decoder‑only fine‑tuning on \~25,000 hours of Japanese data; encoder and codebook are frozen.
• —Compatibility: Because the encoder and codebook are unchanged, speech tokens produced at encode time are identical with the original XCodec2. Any downstream model expecting XCodec2 codes can use Anime‑XCodec2 as a drop‑in decoder (e.g., Llasa).
• —Sampling rate: 16 kHz (XCodec2 operates at 16 kHz).

2) Intended Use

• —Decode XCodec2 speech tokens (e.g., from Llasa or other AR token generators trained on XCodec2 codes) into Japanese speech with improved naturalness for anime/game‑style voices.
• —Reconstruction of Japanese speech from XCodec2 tokens when analyzing or building Japanese‑focused speech pipelines.

3) Limitations & Trade‑offs

• —Language scope: Optimized for Japanese. Performance on other languages may degrade compared to the baseline XCodec2
• —Sampling rate: 16 kHz only (resample inputs to 16 kHz before encoding; decode assumes 16 kHz).
• —Content domain: Tuned toward anime/game‑style voices; out‑of‑domain speech may not benefit.

4) Data (High‑Level)

• —\~25,000 hours of Japanese speech, with a focus on anime/game‑style voices (acting, character voices, etc.).
• —Data preparation included resampling to 16 kHz and standard loudness/peak checks where appropriate.

5) Training Procedure (High‑Level)

• —Updated (fine‑tuned): generator.backbone, generator.head, fc_post_a
• —Frozen: all other non‑listed components

Goal: preserve token compatibility with HKUSTAudio/xcodec2 while improving reconstruction quality for Japanese anime/game‑style speech.

6) Samples (A/B Listening)

Note: original audio is 48 / 44.1 kHz, while reconstructed audios are at 16 kHz

These samples come from [NandemoGHS/Japanese-Eroge-Voice](https://huggingface.co/datasets/NandemoGHS/Japanese-Eroge-Voice) and were not included in the training or validation data.

7) License

• —CC‑BY‑NC 4.0 (same as the original XCodec2 license).
• —See: https://creativecommons.org/licenses/by-nc/4.0/

8) Acknowledgements

• —Original: HKUSTAudio/xcodec2
• —Thanks to contributors and the community around Japanese speech resources.