Research map / Large language models / Language, safety and interpretability
AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing
William Chen, Prem Seetharaman, Rithesh Kumar, Oriol Nieto, Shinji Watanabe, Justin Salamon, Zeyu Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diffusion forcing noise and MLLM representations (Transfusion Forcing) versus plain DiT conditioned on clean input audio (ablation) | audio editing model · StoryGen-Eval editing | editFLAM: 2.7 -> 18.6 | Improved |
| add structured chain-of-thought reasoning to the diffusion LLM (ablation) | AudioChat · StoryGen-Eval editing | delta multiFLAM: 27.8 -> 11.7 | Improved |
| give the storytelling stage access to the input audio (end-to-end) instead of cascading from LLM text (ablation) | AudioChat · StoryGen-Eval editing | delta multiFLAM: 26.7 -> 11.7 | Improved |
| use SCT instead of MoT architecture (ablation) | AudioChat · StoryGen-Eval editing | editFLAM: 9.6 -> 18.6 | Improved |
| use SCT instead of vanilla dense model (ablation) | AudioChat · StoryGen-Eval editing | delta multiFLAM: 15.5 -> 11.7 | Improved |
| use SCT instead of dense or MoT for text-to-audio (ablation) | AudioChat · AuditionSFX T2A | CLAPscore: 65.3 (Dense), 65.8 (MoT), 65.5 (SCT) | No clear effect |
| AudioChat vs WavJourney for storytelling | audio storytelling · StoryGen-Eval storytelling | latency (s): 628 -> 32 | Improved |
| decompose abstract prompt with an LLM before SAO | Stable Audio Open · StoryGen-Eval storytelling | multiFLAM: 35.6 -> 32.2 | Hurt |
| use previous generation instead of ground truth as input in two-turn editing | AudioChat · StoryGen-Eval two-turn editing | delta multiFLAM: 10.5 -> 11.8 | No clear effect |
| evaluate on real AudioCaps audio instead of synthetic | AudioChat · editFLAM | editFLAM: 18.6 -> 15.5 | Hurt |
| replace OpenFLAM with PE-A embedding model in metrics (ablation) | evaluation metrics · StoryGen-Eval editing | method ranking | No clear effect |
Closest papers by meaning
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.88)
- ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.88)
- AudioGenX: Explainability on Text-to-Audio Generative Models (AAAI 2025 · similarity 0.88)
- SAO-Instruct: Free-form Audio Editing using Natural Language Instructions (NeurIPS 2025 · similarity 0.88)
- AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (AAAI 2024 · similarity 0.88)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
