On Oct 26 Tencent Hunyuan (Tencent Hy) open-sourced Hy-SimulMT, a real-time simultaneous interpreting model for bidirectional Chinese-English speech-to-speech translation. It translates while the speaker talks, outputs both translated text and speech, and can clone the speaker's voice in real time. Six checkpoints landed on Hugging Face (30B-A3B translation-only and source-plus-translation Interleave variants, each in BF16, FP8 and GGUF), with the model card listing Apache-2.0; GitHub ships the SimulInfer serving stack and SimulTrain training tools. Vendor-reported: about 3s StreamLAAL on RealSI long-form audio, the lowest among compared systems.
Key Takeaways
- ✓Size and variants: 30B-A3B MoE (about 3B active), six checkpoints: translation-only and Interleave (source plus translation), each in BF16, FP8 and GGUF (Hugging Face collection)
- ✓Latency (vendor-reported): about 3s StreamLAAL on RealSI long-form audio, the lowest among compared systems, with a GEMBA quality score above the compared open models and 4 of 5 commercial APIs
- ✓Short-form (vendor-reported): FLEURS 90.3, CoVoST2 75.5, ACL6060 81.2, RealSI-Sentence 84.7, the last being the top score in the table; Qwen3.8-LiveTranslate scores 88.4/82.1/80.5/84.4 in the same table
- ✓Human comparison: 18 real Chinese and English recordings of about 6 minutes each against 6 CATTI Level 1/2 interpreters; Tencent says quality was on par with often lower latency, and notes the result only covers this small test
- ✓Deployment: one SimulInfer deploy.sh command starts vLLM (>=0.31) plus a web/WebSocket service; speech output and voice cloning need a separate TTS service; llama.cpp support is not yet merged upstream (SimulInfer docs)
Key Decision Metrics at a Glance
Not enough VRAM? Compare cloud API and self-hosting costs
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the problem it targets
Simultaneous interpreting is not just about accuracy: the system has to start speaking before the speaker finishes. Waiting longer improves accuracy but leaves listeners behind. Earlier open options such as SeamlessStreaming were weak on Chinese-English quality and latency, while commercial APIs are hard to self-host. Tencent Hunyuan is open-sourcing the model, serving stack and training tools together, aimed at meetings, talks and interviews.
Architecture and mechanism
Hy-SimulMT is a 30B-A3B MoE model (about 3B active) that consumes a continuous audio stream and incrementally emits translated text and speech, using a speak-first-then-correct strategy to keep lag down. It comes in a translation-only variant and an Interleave variant that also outputs the source text, each in BF16, FP8 and GGUF, and the model card lists Apache-2.0. Real-time voice cloning relies on an external TTS service.
Benchmarks and measured results
All figures are vendor-reported: about 3s StreamLAAL on RealSI long-form audio, the lowest among compared systems; short-form FLEURS 90.3, CoVoST2 75.5, ACL6060 81.2 and RealSI-Sentence 84.7 (top of the table). In the same table Gemini3.5-LiveTranslate leads FLEURS (92.4) and ACL6060 (82.6), and Qwen3.8-LiveTranslate leads CoVoST2 (82.1). The comparison with 6 CATTI interpreters used only 18 recordings, and Tencent says it is not a comprehensive evaluation.
Developer quick start
Clone the GitHub repo, enter SimulInfer and run bash deploy.sh --install-server --model tencent/Hy-SimulMT-30B-A3B-Interleave to start vLLM (>=0.31) and a web/API service on port 8080, with file upload and live microphone over WebSocket. Weights are on Hugging Face; llama.cpp support for the GGUF builds is not merged upstream yet, so wait for an official release. SimulTrain covers continued training.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.