Research map / Video / Video-language understanding
Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding
Rui-Chen Zheng, Wenrui Liu, Hui-Peng Du, Qinglin Zhang, Chong Deng, Qian Chen, Wen Wang, Yang Ai, Zhen-Hua Ling
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use variable-frame-rate clustering (tau 0.7, Smax 4, 30.95 Hz) instead of fixed 40 Hz | WavTokenizer · LibriTTS test-clean reconstruction | UTMOS: 3.6107 -> 3.8949 | Improved |
| variable rate (tau 0.7, Smax 4) vs fixed 40 Hz | WavTokenizer · LibriTTS test-clean reconstruction | PESQ / STOI: PESQ 1.7075 -> 1.7095; STOI 0.8652 -> 0.8601 | No clear effect |
| variable rate (tau 0.8) vs 75 Hz fixed | WavTokenizer · LibriTTS test-clean reconstruction | UTMOS: 4.0247 -> 4.0000 | No clear effect |
| raise similarity threshold tau from 0.6 to 0.8 (ablation) | VARSTok · LibriTTS test-clean reconstruction | UTMOS: 3.83 -> 4.00 (frame rate rises 26.29 -> 36.81 Hz) | Improved |
| increase max span Smax from 2 to 8 (ablation) | VARSTok · LibriTTS test-clean reconstruction | UTMOS / PESQ: UTMOS 4.0379 -> 3.6466; PESQ 2.0694 -> 1.4532 (46.50 -> 22.38 Hz) | Hurt |
| variable-rate tokens vs fixed 40 Hz for semantic probing | WavTokenizer · ARCH AudioMNIST | accuracy: 0.4597 -> 0.6111 | Improved |
| variable-rate tokens vs fixed 40 Hz for semantic probing | WavTokenizer · ARCH RAVDESS | accuracy: 0.2847 -> 0.2951 (tau 0.7); 0.2639 (tau 0.8) | Mixed |
| use VARSTok (tau 0.8) tokens in AR TTS vs 40 Hz WavTokenizer | decoder-only AR TTS · UniCATS zero-shot TTS | WER (%): 7.481 -> 6.787 | Improved |
| use VARSTok (tau 0.8) tokens in AR TTS | decoder-only AR TTS · UniCATS zero-shot TTS | UTMOS: 3.920 -> 4.246 | Improved |
| lower token rate (tau 0.7) in TTS | decoder-only AR TTS · UniCATS zero-shot TTS | speaker similarity: 0.918 -> 0.895 | Hurt |
| lower tau to 0.6 (26.29 Hz) in TTS (ablation) | VARSTok · UniCATS zero-shot TTS | WER (%): 7.294 -> 9.393 | Hurt |
Closest papers by meaning
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.87)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.87)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.87)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.86)
- SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models (ICLR 2024 · similarity 0.86)
- SyllableLM: Learning Coarse Semantic Units for Speech Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
