Research map / Vision-language and multimodal / Multimodal LLMs

PrismAudio: Decomposed Chain-of-Thought and Multi-dimensional Rewards for Video-to-Audio Generation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PrismAudio, which decomposes video-to-audio chain-of-thought into four perceptual CoT modules paired with reward functions, trains with a hybrid ODE-SDE GRPO variant (Fast-GRPO), and introduces the AudioCanvas benchmark.

Problem

Video-to-audio generation must jointly balance semantic consistency, temporal synchrony, aesthetic quality, and spatial accuracy, but monolithic reasoning and single reconstruction losses entangle these objectives and lack human preference alignment.

Approach

Decompose V2A chain-of-thought into Semantic, Temporal, Aesthetic, and Spatial CoT modules, each paired with a dedicated reward (CLAP, Synchformer, Audiobox-Aesthetics, StereoCRW), and optimize them jointly with Fast-GRPO, which applies SDE sampling only within a random window of denoising steps and ODE elsewhere.

Key results

• On the in-domain VGGSound test set, PrismAudio reports CLAP 0.47 vs 0.43 for ThinkSound, DeSync 0.41 vs 0.55, and CRW 7.72 vs 13.47.
• On out-of-domain AudioCanvas, CLAP-based semantic score rises from 0.47 (w/o CoT-RL) to 0.52, and CRW error drops from 15.30 to 12.87 after CoT-RL.
• Fast-GRPO reaches a final semantic reward of about 0.51 versus 0.47 for Flow-GRPO and surpasses Flow-GRPO's final performance in about 200 steps, versus more than 600 for Flow-GRPO.

Surprising findings

• Single-dimension RL rewards degrade other dimensions; the Aesthetic-only model more than doubles the FD (1.90 to 4.50) on AudioCanvas.
• PrismAudio slightly exceeds ground-truth audio on CLAP and DeSync on AudioCanvas, which the authors attribute to metric proxies penalizing natural variation.

Limitations

• The full training pipeline is compute-heavy (16-24 A800 GPUs over 2-3 weeks), and the 1.6x Fast-GRPO speedup is modest relative to full SDE-GRPO.
• Reward models are imperfect proxies, and the authors note ground-truth variation may be penalized by them.
• CoT annotations for AudioCanvas and training rely on Gemini 2.5 Pro, and human verification covers only 200 sampled videos.

Experimental setup

Datasets: VGGSound WavCaps AudioCaps

Benchmarks: VGGSound test set AudioCanvas

Metrics: CLAP DeSync PQ PC CE CU GCC CRW FD KL

Baselines: Frieren V2A-Mapper AudioX HunyuanVideo-Foley MMAudio ThinkSound

Builds on: ThinkSound Flow-GRPO GRPO MMAudio VideoPrism

Closest papers by meaning

• ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.90)
• Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards (ICLR 2026 · similarity 0.88)
• V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.88)
• From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
• Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.88)
• Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.87)