Modelpublic

lugan/SynTTS-Commands-Media-Benchmarks

sourceHugging Facemitupdated 8mo agoView on Hugging Face

2likes90downloads

Model Card

SynTTS Commands Media Benchmarks

![Dataset](https://huggingface.co/datasets/lugan/SynTTS-Commands-Media-Dataset) ![Code](https://github.com/lugan113/SynTTS-Commands-Official) ![License](https://opensource.org/licenses/MIT)

🚀 Project Navigation

Welcome to the official model repository for the paper "SynTTS-Commands". Here you can find the pre-trained checkpoints for KWS tasks.

—📄 Paper: Read the detailed technical report on arXiv.
—💾 Dataset: Download the training data at SynTTS-Commands-Media-Dataset.
—💻 Code: Access training scripts and inference code on GitHub.

📈 Benchmark Results and Analysis

We present a comprehensive benchmark of six representative acoustic models on the SynTTS-Commands-Media Dataset across both English (EN) and Chinese (ZH) subsets. All models are evaluated in terms of classification accuracy, cross-entropy loss, and parameter count, providing insights into the trade-offs between performance and model complexity in multilingual voice command recognition.

Performance Summary

Model	EN Loss	EN Accuracy	EN Params	ZH Loss	ZH Accuracy	ZH Params
MicroCNN	0.2304	93.22%	4,189	0.5579	80.14%	4,255
DS-CNN	0.0166	99.46%	30,103	0.0677	97.18%	30,361
TC-ResNet	0.0347	98.87%	68,431	0.0884	96.56%	68,561
CRNN	0.0163	99.50%	1.08M	0.0636	97.42%	1.08M
MobileNet-V1	0.0167	99.50%	2.65M	0.0552	97.92%	2.65M
EfficientNet	0.0182	99.41%	4.72M	0.0701	97.93%	4.72M

🔍 Key Findings

Our results demonstrate that the SynTTS-Commands dataset supports high-accuracy command recognition in both languages. Notably, the top-performing models achieve over 99.4% accuracy on English and nearly 98% on Chinese, confirming the dataset’s quality and suitability for real-world deployment.

—Top Performers: Among all models, CRNN attains the best English accuracy (99.50%) and the lowest loss (0.0163). MobileNet-V1 yields the lowest loss on Chinese (0.0552) and competitive English performance (matching CRNN’s 99.50% accuracy). Interestingly, EfficientNet shows slightly higher Chinese accuracy (97.93%) than MobileNet-V1, suggesting better calibration or robustness despite a higher loss.

—Accuracy-Complexity Trade-off: Lightweight models exhibit a clear trade-off. MicroCNN, with only ~4.2K parameters, achieves 93.22% accuracy on English but drops to 80.14% on Chinese, highlighting the increased difficulty of modeling tonal and phonetic richness in Mandarin with ultra-compact architectures. DS-CNN and TC-ResNet, with under 70K parameters, already recover strong performance (>96.5% in both languages), underscoring their efficiency for resource-constrained applications.

Overall, the benchmark establishes strong baselines across a wide spectrum of model scales—from ultra-light MicroCNN to modern EfficientNet—demonstrating that moderate-complexity models can deliver near-SOTA performance suitable for edge deployment.