Research map / Video / Video segmentation and tracking
FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
Jiaqi Li, Yao Qian, Yuxuan Hu, leying zhang, Xiaofei Wang, Heng Lu, Manthan Thakker, Jinyu Li, sheng zhao, Zhizheng Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DualCodec SSL feature (w2v-bert-2) with SenseVoice ASR feature (ablation) | FlexiCodec-style codec at fixed 6.25Hz · LibriSpeech test-clean, RVQ1 | WER (%): 31.5 -> 5.99 | Improved |
| add dynamic frame rate merging (ablation) | FlexiCodec with transformers and FSQ · LibriSpeech test-clean 6.25Hz, RVQ1 | WER (%): 5.22 -> 4.15 | Improved |
| add Merging and Unmerging transformers to fixed-rate codec (ablation) | fixed-rate SenseVoice codec · 6.25Hz, RVQ1 | WER (%): 5.99 -> 5.40 | Improved |
| use FSQ instead of VQ for semantic tokens (ablation) | FlexiCodec · 6.25Hz, RVQ1 | WER (%): 4.43 -> 4.15 | Improved |
| add Transformer to frame merging or unmerging modules (ablation) | FlexiCodec · 6.25Hz acoustic test | PESQ/UTMOS/SIM: PESQ 2.46 -> 2.76 (merging) | Improved |
| use separate parameters for the two merging streams instead of shared (ablation) | FlexiCodec frame merging · 6.25Hz, RVQ1 | WER (%): 8.24 -> 4.15 | Improved |
| use dynamic frame rate instead of fixed frame rate (ablation) | FlexiCodec · 6.25Hz, RVQ1/ASR probing/RVQ1:8 | WER: relative 26% (RVQ-1), 21% (ASR probing), 8% (RVQ1:8) | Improved |
| use dynamic frame rate instead of fixed frame rate (ablation) | FlexiCodec · acoustic metrics (PESQ, UTMOS, MCD, SIM) | PESQ/UTMOS | No clear effect |
| use Whisper-medium or averaged SenseVoice instead of last-layer SenseVoice-Small (ablation) | FlexiCodec · 6.25Hz, RVQ1 | WER (%): 4.15 -> 9.83 (Whisper), 4.66 (avg SenseVoice) | Hurt |
| use 12.5Hz NAR instead of 50Hz Mel NAR in TTS (ablation) | FlexiCodec-TTS · LibriSpeech-PC test-clean | WER (%): 3.2 -> 4.6 at 6.25Hz AR | Hurt |
| lower AR frame rate from 12.5Hz to 6.25Hz with 50Hz NAR (ablation) | FlexiCodec-TTS · LibriSpeech-PC test-clean | WER (%): 2.5 -> 3.2 | Mixed |
| lower frame rate of retrained codec from 12.5Hz to 6.25Hz (ablation) | DualCodec · LibriSpeech test-clean, RVQ1 | WER (%): 5.93 -> 31.5 | Hurt |
Closest papers by meaning
- FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks (NeurIPS 2025 · similarity 0.90)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.87)
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.87)
- Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.87)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.87)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
