Research map / Architectures and efficiency / Attention and transformers
ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ChA-MAEViT, an MAE-based multi-channel vision transformer that uses dynamic channel-patch masking, memory tokens, hybrid token fusion, and a channel-aware decoder to improve cross-channel learning on CHAMMI, JUMP-CP, and So2Sat.
Problem
Standard MAEs rely on cross-channel redundancy via random patch masking, which fails for multi-channel imaging (MCI) where channels carry complementary information with little overlap, so models learn mostly within-channel structure and handle missing channels poorly.
Approach
Combine patch and channel-level masking (dynamic channel-patch masking) so missing whole channels are reconstructed as supervision, add learnable memory tokens for cross-channel information sharing, fuse CLS with patch tokens for pooling, and use a single shared lightweight decoder conditioned on channel tokens.
Key results
- ChA-MAEViT reaches 74.63 average F1 on CHAMMI and 90.73% full-channel accuracy on JUMP-CP, versus 69.77 and 69.19 for the best prior model DiChaViT.
- On So2Sat full-channel accuracy, ChA-MAEViT achieves 67.44% vs 63.36% for DiChaViT; the paper reports gains of 3.0-21.5% over prior MCI-ViTs.
- Ablation shows removing DCP masking causes the largest drop, e.g., JUMP-CP partial accuracy falls from 68.05 to 52.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ChA-MAEViT instead of DiChaViT | MCI vision models · CHAMMI | Avg score: 69.77 -> 74.63 | Helped |
| use ChA-MAEViT instead of CA-MAE + supervised loss | MCI vision models · JUMP-CP Full | accuracy: 69.54 -> 90.73 | Helped |
| use ChA-MAEViT instead of DiChaViT | MCI vision models · JUMP-CP Partial | accuracy: 57.98 -> 68.05 | Helped |
| use ChA-MAEViT instead of DiChaViT | MCI vision models · So2Sat Full | accuracy: 63.36 -> 67.44 | Helped |
| use ChA-MAEViT instead of DiChaViT | MCI vision models · So2Sat Partial | accuracy: 47.76 -> 52.11 | Helped |
| add DCP Masking instead of random patch masking (ablation) | ChA-MAEViT · JUMP-CP Partial | accuracy: 52.33 -> 68.05 | Helped |
| add DCP Masking instead of random patch masking (ablation) | ChA-MAEViT · So2Sat Partial | accuracy: 28.70 -> 52.11 | Helped |
| add Hybrid Token Fusion instead of using only the [CLS] token (ablation) | ChA-MAEViT · JUMP-CP Full | accuracy: 88.25 -> 90.73 | Helped |
| add Memory Tokens (ablation) | ChA-MAEViT · JUMP-CP Full | accuracy: 87.81 -> 90.73 | Helped |
| add Channel-Aware Decoder instead of separate decoders (ablation) | ChA-MAEViT · JUMP-CP Full | accuracy: 87.52 -> 90.73 | Helped |
| use DCP Alternate rather than DCP Combination (ablation) | ChA-MAEViT · JUMP-CP Full | accuracy: 85.95 -> 90.73 | Helped |
| use DCP Alternate rather than DCP Combination (ablation) | ChA-MAEViT · So2Sat Full | accuracy: 67.44 -> 66.47 | Hurt |
Surprising findings
- Fixed-ratio channel masking performs poorly on CHAMMI (47.97 avg) and channel+patch fixed masking is even worse on that benchmark, indicating naive masking can hurt.
- Random patch masking performs well with full channels but drops sharply with partial channels (So2Sat partial 28.70).
Limitations
- Evaluated only on microscopy and satellite datasets; generalization to other MCI domains such as volumetric medical imaging is untested.
- Adapting to unseen channels is not thoroughly evaluated, and the method requires extra hyperparameter tuning.
- Several hyperparameters (masking probabilities, patch ratios) are set per dataset, adding tuning cost.
Experimental setup
Datasets: CHAMMI JUMP-CP So2Sat 38-Cloud
Benchmarks: CHAMMI JUMP-CP So2Sat 38-Cloud
Metrics: F1 score top-1 accuracy accuracy IoU precision recall
Baselines: HyperNetViT DepthwiseViT TemplateMixingViT CA-MAE ChAda-ViT ChannelViT DiChaViT SimCLR SimSiam iBOT DINOv2 MAE
Builds on: Masked Autoencoders CA-MAE DiChaViT ChannelViT Hierarchical Channel Sampling register tokens
Closest papers by meaning
- DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images (NeurIPS 2026 · similarity 0.89)
- Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers (NeurIPS 2024 · similarity 0.89)
- Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning (AAAI 2025 · similarity 0.88)
- Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding (NeurIPS 2026 · similarity 0.87)
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (ICLR 2024 · similarity 0.87)
- Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.