Research map / Architectures and efficiency / Attention and transformers
DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DC-ViT, a multi-channel vision transformer that decouples self-attention into spatial and channel-wise pathways (DSA) and uses hierarchical per-channel-then-cross-channel pooling (DAG), improving existing MC-ViTs on CHAMMI, JUMP-CP and So2Sat.
Problem
Joint self-attention over all channel tokens in multi-channel vision transformers entangles spatial and cross-channel interactions, diluting channel-specific semantics in multi-channel imaging data with heterogeneous channel configurations.
Approach
Decompose token updates into spatial (intra-channel) attention applied at all layers and channel-wise attention applied only at selected middle layers with a learnable mixing scalar, plus a hierarchical pooling that first pools within channels and then across channels with learned channel importance.
Key results
- On CHAMMI, DC-ViT raises ChannelViT average from 64.90 to 71.72 when trained from scratch.
- With ChaMAEViT, DC-ViT improves CHAMMI average from 73.13 to 76.33 and JUMP-CP full from 85.23 to 86.11.
- Ablation: removing DSA drops CHAMMI average from 71.72 to 70.02 and removing both gives 62.91.
Surprising findings
- Applying channel attention in early layers reduces performance, while placing it in middle layers is best.
- DC-ViT degrades ChAdaViT under partial-channel evaluation (42.15 to 25.09 on JUMP-CP partial).
Limitations
- Inter-channel attention requires one-to-one spatial correspondence between channels, making random patch sampling infeasible.
- Degrades ChAdaViT under partial-channel inference.
Experimental setup
Datasets: CHAMMI JUMP-CP So2Sat HPAv23
Benchmarks: CHAMMI JUMP-CP So2Sat
Metrics: F1 score accuracy
Baselines: ChAdaViT ChannelViT DiChaViT ChaMAEViT
Builds on: ChAdaViT ChannelViT DiChaViT ChaMAEViT Vision Transformer iBOT
Closest papers by meaning
- Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers (NeurIPS 2024 · similarity 0.90)
- ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning (NeurIPS 2025 · similarity 0.89)
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (ICLR 2024 · similarity 0.89)
- ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images (CVPR 2024 · similarity 0.87)
- CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction (CVPR 2025 · similarity 0.86)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.