AI Paper Atlas

Research map / Image generation / Autoregressive and latent generation

Direct Conditioning of Audio Diffusion Transformers on fMRI Reveals Cortical Contributions to Sound Reconstruction

Matteo Ciferri, Tonio Weidler, Matteo Ferrante, Nicola Toschi, Elia Formisano

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: Diffusion audio synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes directly conditioning an audio diffusion transformer on fMRI brain activity to reconstruct sound, rather than relying only on text-based conditioning.

Problem

Reconstructing audio from brain activity is hard because audio diffusion models are built for text conditioning and lack a mechanism for continuous non-linguistic neural signals like fMRI responses.

Approach

Extends direct brain-activity conditioning, previously shown in the visual domain, to audio diffusion transformers so that fMRI responses feed the diffusion model directly as a continuous conditioning signal.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune cross-attention layers of the DiT backbone with the adapter (ablation)MAAS · fMRI sound reconstructionpairwise ID accuracy: 73.1% -> 85.4%Improved
route brain tokens through dedicated adapter instead of brain-plus-empty residual (ablation)MAAS · fMRI sound reconstructionpairwise ID accuracy: 63.1% -> 85.4%Improved
keep empty text prompt and add brain pathway rather than replacing conditioning (brain-only) (ablation)MAAS · fMRI sound reconstructionCLAP similarity: 0.403 -> 0.614Improved
condition directly on fMRI instead of two-stage fMRI-to-latent predictionMAAS vs two-stage · fMRI sound reconstructionCLAP similarity: 0.509 -> 0.614Improved
add oracle captions at inference to MAAS (zero-shot)MAAS · fMRI sound reconstructionpairwise ID accuracy: 85.4% -> 84.6%No clear effect
condition on caption only instead of brain activityStable Audio · fMRI sound reconstructionMel-SSIM: 0.297 -> 0.166Hurt
shuffle brain-stimulus pairs within category during training (ablation)MAAS · pairwise IDaccuracy: 85.4% -> 73.6%Hurt
shuffle brain-stimulus pairs without restriction (ablation)MAAS · pairwise IDaccuracy: 85.4% -> 53.4%Hurt
remove a single ROI from MAAS input (ablation)MAAS · NSL fidelityrelative fidelity: -2.8% to +1.7% per ROINo clear effect
use only a single ROI instead of all six (ablation)MAAS · NSL fidelityrelative fidelity: up to -83% in individual foldsHurt
remove any ROI for non-speech vocalizations (ablation)MAAS · non-speech vocalizationsNSL fidelity: +4% to +12% on averageMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy