Research map / Architectures and efficiency / Attention and transformers
Disentangling Channel Semantics in Vision Transformers via Token Decorrelation and Composition-Aware Modulation
Daeun Kim, Hyejin Park, Hyesong Choi, Dongbo Min
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multi-channel Vision Transformer that disentangles channel semantics via token decorrelation and composition-aware modulation, building on memory-token approaches that augment the [CLS] token.
Problem
Multi-channel images have heterogeneous channel semantics over a shared spatial structure; existing multi-channel ViTs that augment [CLS] with memory tokens struggle to jointly capture globally shared structure and channel-specific variation under varying channel compositions.
Approach
Decorrelate tokens to separate channel-specific from shared information and use composition-aware modulation so representations adapt to which channels are present, rather than relying only on memory tokens appended to [CLS].
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FTL to ChA-MAEViT (ablation) | ChA-MAEViT · JUMP-CP (all channels) | accuracy: 90.35 -> 91.58 | Improved |
| add FTL to ChA-MAEViT (ablation) | ChA-MAEViT · JUMP-CP (partial channels) | accuracy: 68.10 -> 68.23 | No clear effect |
| add CAM to ChA-MAEViT (ablation) | ChA-MAEViT · JUMP-CP (all channels) | accuracy: 90.35 -> 92.46 | Improved |
| add both FTL and CAM (ablation) | ChA-MAEViT · JUMP-CP (all channels) | accuracy: 90.35 -> 94.35 | Improved |
| condition CAM on FTL-refined CLS rather than raw CLS (ablation) | CAM · JUMP-CP (all channels) | accuracy: 93.18 -> 95.41 | Improved |
| condition CAM on CLS instead of external channel multi-hot vector (ablation) | CAM · JUMP-CP (all channels) | accuracy: 92.57 -> 95.41 | Improved |
| use asymmetric weight in FTL instead of equal weight (ablation) | FTL · JUMP-CP (all channels) | accuracy: 94.09 -> 95.41 | Improved |
| apply CAM in last two blocks instead of first two (ablation) | CAM · JUMP-CP (all channels) | accuracy: 92.18 -> 95.41 | Improved |
| add FTL+CAM to IC-ViT mem | IC-ViT mem · JUMP-CP (all channels) | accuracy: 86.68 -> 87.83 | Improved |
| add FTL+CAM to DiChaViT mem | DiChaViT mem · JUMP-CP (all channels) | accuracy: 64.83 -> 68.09 | Improved |
| use MuCa-ViT at ViT-Base scale on CHAMMI | MuCa-ViT · CHAMMI | avg score: 76.20 -> 76.72 | Improved |
| use MuCa-ViT versus ChA-MAEViT on So2Sat partial channels | MuCa-ViT · So2Sat (partial channels) | accuracy: 49.83 -> 52.12 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- ChA-MAEViT — ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning (NeurIPS 2025) · better on 5 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Surprising findings
- Abstract indicates that memory-token augmentation of [CLS] has a limitation under varying channel compositions, though the specific finding is truncated in the available text.
Experimental setup
Baselines: multi-channel Vision Transformers with memory tokens
Builds on: memory tokens ChannelViT
Closest papers by meaning
- ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning (NeurIPS 2025 · similarity 0.87)
- DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images (NeurIPS 2026 · similarity 0.87)
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (ICLR 2024 · similarity 0.86)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.86)
- Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers (NeurIPS 2024 · similarity 0.85)
- Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
