Research map / Image generation / Autoregressive and latent generation
FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds GOAT, a 4.54-hour annotated Gezi Opera audio-text dataset with new vowel-morphing and vocal-run annotation strategies, and proposes FT-GAN, a GAN-based acoustic model with a pitch encoder, Conformer mel predictor, and speech pre-training on Hokkien speech for Chinese opera singing synthesis.
Problem
Chinese opera synthesis is rarely studied due to scarce high-quality annotated opera data and the fine-grained, highly expressive pitch and duration patterns (slurs, long vowels) that existing singing voice synthesis models fail to model.
Approach
Combines opera-specific annotation strategies (vowel morphing, vocal run, pseudo pitch extraction) with an acoustic GAN that adds a separate pitch encoder, FiLM fusion, Conformer mel predictor, and multi-length/sub-frequency discriminators, plus pre-training on Hokkien speech treated as opera-like data.
Key results
- On Gezi Opera, FT-GAN reached 4.22 expressiveness and 4.18 naturalness MOS, a 0.28 naturalness gain over the strongest baseline VISinger2 (3.90).
- Ablations show removing vowel morphing drops pronunciation CMOS by 0.43 and removing the pitch encoder drops expressiveness by 0.37.
- On Peking Opera (1.7h), FT-GAN achieved 3.70 naturalness MOS, 0.4 above the strongest baseline FastSpeech2 (3.30).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FT-GAN fine-grained tune modeling | Gezi Opera synthesis · Gezi Opera | naturalness MOS: 3.90 ± 0.04 -> 4.18 ± 0.03 | Helped |
| add FT-GAN fine-grained tune modeling | Peking Opera synthesis · Peking Opera | naturalness MOS: 3.30 ± 0.05 -> 3.70 ± 0.06 | Helped |
| add Hokkien speech pre-training (ablation) | FT-GAN · Gezi Opera | naturalness MOS: 4.08 ± 0.03 -> 4.18 ± 0.03 | Helped |
| add Hokkien speech pre-training (ablation) | FT-GAN · Gezi Opera | pronunciation MOS: 4.46 ± 0.03 -> 4.40 ± 0.03 | Hurt |
| add vowel morphing annotation (ablation) | FT-GAN training data · Gezi Opera | pronunciation CMOS: +0.43 | Helped |
| add vocal run annotation (ablation) | FT-GAN training data · Gezi Opera | consistency CMOS: +0.32 | Helped |
| add a pitch encoder (ablation) | FT-GAN · Gezi Opera | naturalness CMOS: +0.34 | Helped |
| add a discriminator (ablation) | FT-GAN · Gezi Opera | naturalness CMOS: +0.23 | Helped |
| replace the mel-predictor FFT block with Conformer (ablation) | FT-GAN · Gezi Opera | naturalness CMOS: +0.32 | Helped |
Surprising findings
- Speech pre-training improved naturalness but slightly lowered pronunciation accuracy (Pro 4.40 vs 4.46 without it).
- Without the discriminators, consistency rose (+0.08) even though pronunciation, naturalness and expressiveness dropped.
Limitations
- Due to insufficient training data, the model produces electric and mute sounds for extremely high or low pitches or rapidly alternating pitch.
- Audio sampling rate limited to 24kHz; 48kHz opera synthesis left to future work.
- Peking Opera experiments required switching from note input to ground-truth f0 input because note-input models failed.
Experimental setup
Datasets: GOAT (Gezi Opera audio-text dataset) Hokkien speech crawled from the Internet Peking Opera dataset (Gong et al. 2017)
Benchmarks: GOAT Gezi Opera test Peking Opera synthesis
Metrics: MOS pronunciation MOS naturalness MOS expressiveness CMOS consistency
Baselines: FastSpeech2 DiffSinger VISinger2 DurIAN
Builds on: HifiSinger FastSpeech2 VITS PatchGAN HifiGAN
Closest papers by meaning
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.85)
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks (NeurIPS 2024 · similarity 0.84)
- SongSong: A Time Phonograph for Chinese SongCi Music from Thousand of Years Away (AAAI 2025 · similarity 0.83)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.83)
- StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis (AAAI 2024 · similarity 0.83)
- TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.