Research map / Image generation / Autoregressive and latent generation
Direct Conditioning of Audio Diffusion Transformers on fMRI Reveals Cortical Contributions to Sound Reconstruction
Matteo Ciferri, Tonio Weidler, Matteo Ferrante, Nicola Toschi, Elia Formisano
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes directly conditioning an audio diffusion transformer on fMRI brain activity to reconstruct sound, rather than relying only on text-based conditioning.
Problem
Reconstructing audio from brain activity is hard because audio diffusion models are built for text conditioning and lack a mechanism for continuous non-linguistic neural signals like fMRI responses.
Approach
Extends direct brain-activity conditioning, previously shown in the visual domain, to audio diffusion transformers so that fMRI responses feed the diffusion model directly as a continuous conditioning signal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune cross-attention layers of the DiT backbone with the adapter (ablation) | MAAS · fMRI sound reconstruction | pairwise ID accuracy: 73.1% -> 85.4% | Improved |
| route brain tokens through dedicated adapter instead of brain-plus-empty residual (ablation) | MAAS · fMRI sound reconstruction | pairwise ID accuracy: 63.1% -> 85.4% | Improved |
| keep empty text prompt and add brain pathway rather than replacing conditioning (brain-only) (ablation) | MAAS · fMRI sound reconstruction | CLAP similarity: 0.403 -> 0.614 | Improved |
| condition directly on fMRI instead of two-stage fMRI-to-latent prediction | MAAS vs two-stage · fMRI sound reconstruction | CLAP similarity: 0.509 -> 0.614 | Improved |
| add oracle captions at inference to MAAS (zero-shot) | MAAS · fMRI sound reconstruction | pairwise ID accuracy: 85.4% -> 84.6% | No clear effect |
| condition on caption only instead of brain activity | Stable Audio · fMRI sound reconstruction | Mel-SSIM: 0.297 -> 0.166 | Hurt |
| shuffle brain-stimulus pairs within category during training (ablation) | MAAS · pairwise ID | accuracy: 85.4% -> 73.6% | Hurt |
| shuffle brain-stimulus pairs without restriction (ablation) | MAAS · pairwise ID | accuracy: 85.4% -> 53.4% | Hurt |
| remove a single ROI from MAAS input (ablation) | MAAS · NSL fidelity | relative fidelity: -2.8% to +1.7% per ROI | No clear effect |
| use only a single ROI instead of all six (ablation) | MAAS · NSL fidelity | relative fidelity: up to -83% in individual folds | Hurt |
| remove any ROI for non-speech vocalizations (ablation) | MAAS · non-speech vocalizations | NSL fidelity: +4% to +12% on average | Mixed |
Closest papers by meaning
- MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding (NeurIPS 2026 · similarity 0.87)
- AND: Audio Network Dissection for Interpreting Deep Acoustic Models (ICML 2024 · similarity 0.87)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.87)
- Towards Homogeneous Lexical Tone Decoding from Heterogeneous Intracranial Recordings (ICLR 2025 · similarity 0.86)
- The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · similarity 0.85)
- ZeroSep: Separate Anything in Audio with Zero Training (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
