Research map / Vision-language and multimodal / Cross-modal learning
RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a self-reinforcing modality-bias cycle that deactivates attention's dynamic fusion in multimodal Transformers and proposes RollingQ, which rotates the [class] query toward a rebalanced anchor to restore adaptive modality weighting.
Problem
Dynamic fusion via self-attention in multimodal Transformers loses its ability to adaptively weight modalities; the model over-attends to one modality regardless of its informativeness, underperforming static fusion on audio-visual and multimodal sentiment tasks.
Approach
Measure the attention imbalance between modalities with an Attention Imbalance Rate (AIR) and, when it is large, rotate the [class] query toward a rebalanced anchor (weighted toward the unbiased modality's average key) via an SVD-derived rotation matrix, breaking the feedback loop between attention allocation and encoder optimization.
Key results
- On Kinetic-Sound, Vanilla MT+RollingQ reaches 69.3% vs 67.0% for vanilla MT; on CREMA-D it reaches 51.9% vs 48.8%.
- Pearson correlation between attention and noise-input indicator on Kinetic-Sound rises from 0.44 (Vanilla MT) to 0.78 with RollingQ.
- RollingQ adds about 1% parameters and 0.1% GFLOPs over vanilla MT on CREMA-D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RollingQ to Vanilla MT | Multimodal Transformer · CREMA-D | accuracy: 48.8 -> 51.9 | Helped |
| Apply RollingQ to Vanilla MT | Multimodal Transformer · Kinetic-Sound | accuracy: 67.0 -> 69.3 | Helped |
| Apply RollingQ to multimodal fusion | CMU-MOSEI (V+T) Transformer · CMU-MOSEI (V+T) | accuracy: 62.7 -> 63.2 | Helped |
| Apply RollingQ to MMML | MMML · CREMA-D | accuracy: 52.0 -> 52.7 | Helped |
| Apply RollingQ to MMML | MMML · Kinetic-Sound | accuracy: 69.8 -> 70.7 | Helped |
| Apply RollingQ under Gaussian noise perturbation of audio | Vanilla MT · Kinetic-Sound, noise level 1.00 | accuracy: 34.7 -> 40.6 | Helped |
| Use different batch sizes with RollingQ (ablation) | Vanilla MT+RollingQ · CREMA-D, batch sizes 16, 64, 256 | accuracy: 49.1 -> 50.6; 48.8 -> 51.9; 47.8 -> 48.5 | Mixed |
| Vary encoder depth with RollingQ (ablation) | Vanilla MT+RollingQ · Kinetic-Sound, 2, 4, and 6 encoder layers | accuracy: 57.6 -> 58.5; 67.0 -> 69.3; 73.6 -> 74.6 | Helped |
| Add RollingQ while masking the audio modality (ablation) | Vanilla MT · Kinetic-Sound, audio masked | accuracy: 40.3 -> 42.1 | Helped |
| Add RollingQ while masking the visual modality (ablation) | Vanilla MT · Kinetic-Sound, visual masked | accuracy: 46.6 -> 47.4 | Helped |
| Use RollingQ with QUAG operations (ablation) | Transformer fusion baseline · Kinetic-Sound, QUAG unimodal, crossmodal, avg-audio, and avg-visual | accuracy: 67.8 -> 63.8; 65.5 -> 63.8; 64.8 -> 62.9; 65.6 -> 62.1 | Mixed |
| Move fusion to earlier transformer blocks while using RollingQ (ablation) | Vanilla MT · CREMA-D, 2nd-, 3rd-, and 4th-layer fusion | accuracy: 42.3 -> 42.5; 41.1 -> 42.2; 41.4 -> 43.7 | Helped |
Surprising findings
- Dynamic self-attention fusion underperforms simple static concatenation on Kinetic-Sound (67.0 vs 68.0).
- The attention layer assigns high attention to Gaussian-noise audio inputs, indicating loss of input-adaptive weighting.
Limitations
- Theoretical analysis covers only a single attention layer; multi-layer extension is heuristic (progressive training, rotation limits).
- RollingQ does not directly improve unimodal encoder feature quality.
- Gains on some settings are small (e.g., +0.4 to +0.5 on CMU-MOSEI).
Experimental setup
Datasets: CREMA-D Kinetic-Sound CMU-MOSEI UCF-101 HMDB51
Benchmarks: CREMA-D Kinetic-Sound CMU-MOSEI (V+T) HMDB51 OOD UCF-101 OOD
Metrics: accuracy FRR95 AUROC IDAcc Pearson correlation
Baselines: Concat OGM PMR Vanilla MT MulT MBT JMT MMML unimodal models
Builds on: OGM-GE PMR MBT MulT QUAG attention
Closest papers by meaning
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion (ICML 2024 · similarity 0.86)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.86)
- Understanding Differential Transformer Unchains Pretrained Self-Attentions (NeurIPS 2025 · similarity 0.86)
- JoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention (ICLR 2024 · similarity 0.86)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.85)
- Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.