Research map / Architectures and efficiency / Attention and transformers

When Attention Collapses: Residual Evidence Modeling for Compositional Inference

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Attention head mechanisms · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies slot collapse under additive superposition in slot attention and proposes evidence depletion (sequential slot attention with multiplicative residual evidence and attention bias), which reduces collapse on synthetic, FUSS audio, and LISA gravitational-wave inference.

Problem

Attention-based slot decomposition collapses when multiple sources contribute additively to every token, so multiple slots converge to the same dominant component and weaker sources go unrepresented.

Approach

Track residual explanatory evidence per token and multiplicatively deplete it after each slot attends, with a log-evidence attention bias and evidence-scaled inputs, so later slots are steered toward unexplained tokens without modifying the signal itself.

Key results

• Linear evidence depletion reduces peak overlap collapse on synthetic tasks to 0.07/0.03/0.01 versus 0.32/0.26/0.22 for vanilla slot attention.
• On FUSS, evidence linear depletion reduces collapse from 0.29 (vanilla) to 0.05, while sequential SA collapses to 0.93.
• On LISA ablation, evidence depletion reaches overlap 0.08 vs 0.99 for vanilla SA and flow NLL -6.0 vs +7.1.

Surprising findings

• Quadratic depletion, the form used for LISA, is effectively inert on FUSS (0.31 vs 0.29 for vanilla), while linear depletion works.
• Sequential processing alone worsens collapse on synthetic and FUSS benchmarks relative to vanilla slot attention.

Limitations

• Optimal depletion form and hyperparameters are task-dependent and chosen empirically.
• Current formulation operates at a single level of granularity and may need hierarchical decomposition for complex settings.
• LISA evaluation uses simulated data only; frequency derivative posterior is miscalibrated due to encoder compression.

Experimental setup

Datasets: FUSS synthetic sinusoids, Gaussians, multi-scale mixtures simulated LISA galactic binary data (JaxGB, SciRDv1 noise)

Benchmarks: synthetic sinusoids synthetic Gaussians synthetic multi-scale FUSS LISA galactic binary inference

Metrics: peak overlap rate max active overlap flow NLL CRPS AngCorr coverage

Baselines: vanilla slot attention sequential slot attention loss-based regularization (diversity and orthogonality losses) data-space depletion

Builds on: slot attention DETR clean matching pursuit coverage in neural machine translation

Closest papers by meaning

• Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs (ICLR 2026 · similarity 0.84)
• CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models (NeurIPS 2026 · similarity 0.84)
• The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design (ICML 2026 · similarity 0.84)
• Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs (NeurIPS 2026 · similarity 0.83)
• See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.83)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.83)