Research map / Probabilistic and scientific modeling / Variational and generative sampling
CSSinger: End-to-End Chunkwise Streaming Singing Voice Synthesis System Based on Conditional Variational Autoencoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSSinger, an end-to-end conditional-VAE singing voice synthesizer with chunkwise streaming (semi- and fully-streaming variants), using Natural Padding for a causal HiFi-GAN and a ChunkStream Decoder built on Emformer-style attention.
Problem
Existing end-to-end singing voice synthesis systems use parallel computation over full sequences, which is costly on long sequences and has high latency for real-time or edge deployment, while latent-representation vocoders perform poorly when made causal for streaming.
Approach
Builds on SiFiSinger's conditional VAE and makes the whole pipeline streaming: a Natural Padding strategy that uses real latent values as causal padding in a HiFi-GAN generator, plus a chunk-based ChunkStream Decoder with Emformer-style memory/left/right context and a Causal Smooth Layer, so latency and compute scale per chunk.
Key results
- On Opencpop, CSSinger-FS achieves MOS 3.607±0.091 vs SiFiSinger 3.510±0.097 and CSSinger-SS 2.415±0.092, with the best F0 RMSE (28.601) and MCD (6.715).
- On GPU, CSSinger-FS reduces latency to 0.051 s from 0.180 s for SiFiSinger; on limited CPU it gives 0.965 s latency vs 4.058 s, but SiFiSinger keeps the lowest RTF on GPU (0.038 vs 0.099).
- In TTS on Baker, CSSinger-FS reaches MOS 3.828±0.071 vs SiFiSinger 3.911±0.088, a gap not significant given overlapping confidence intervals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use fully streaming chunkwise decoding and waveform generation instead of parallel inference | CSSinger-FS · Opencpop test | MOS: 3.510±0.097 -> 3.607±0.091 | Helped |
| add Natural Padding to the streaming HiFi-GAN generator | CSSinger · Opencpop test | MOS: 2.415±0.092 -> 3.165±0.095 | Helped |
| use fully streaming chunkwise decoding and waveform generation instead of parallel inference | CSSinger-FS · PopCS | F0 Corr: 0.872 -> 0.885 | Helped |
| use fully streaming chunkwise decoding and waveform generation instead of parallel inference | CSSinger-FS · Baker TTS | MOS: 3.911±0.088 -> 3.828±0.071 | Hurt |
| replace the causal posterior encoder with a non-causal posterior encoder (ablation) | CSSinger-FS · Opencpop ablation | F0 RMSE: 28.101 -> 28.334 | Hurt |
| remove Natural Padding (ablation) | CSSinger-FS · Opencpop ablation | F0 RMSE: 28.101 -> 30.519 | Hurt |
| remove the Causal Smooth Layer (ablation) | CSSinger-FS · Opencpop ablation | F0 RMSE: 28.101 -> 37.872 | Hurt |
Surprising findings
- The semi-streaming variant without Natural Padding performs poorly on Opencpop (MOS 2.415), showing constant one-sided padding is harmful for latent-based vocoders.
- The fully streaming model has higher RTF than the parallel SiFiSinger baseline on GPU and CPU, despite much lower latency.
Limitations
- Fully streaming RTF is worse than the parallel baseline on GPU and CPU
- TTS gains over SiFiSinger are not statistically significant
- Evaluated on a single female singer datasets and a single TTS dataset
Experimental setup
Datasets: Opencpop PopCS Baker
Benchmarks: Opencpop PopCS Baker
Metrics: MOS CMOS F0 RMSE F0 correlation U/UV error MSE MCD latency RTF
Baselines: SiFiSinger VISinger2 DiffSinger CSSinger-SS CSSinger-SS-NP
Builds on: SiFiSinger VITS HiFi-GAN Emformer AudioDec
Closest papers by meaning
- StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis (AAAI 2024 · similarity 0.87)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.86)
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks (NeurIPS 2024 · similarity 0.86)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.86)
- DFlow: A Generative Model Combining Denoising AutoEncoder and Normalizing Flow for High Fidelity Waveform Generation (ICML 2024 · similarity 0.85)
- TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.