Research map / Architectures and efficiency / Attention and transformers
Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ChannelViT, which tokenizes each input channel separately with a learnable channel embedding, and Hierarchical Channel Sampling (HCS), a regularizer that samples channel counts uniformly, to make ViTs robust to partial or varying channel sets in multi-channel imaging.
Problem
Standard ViTs collapse all channels of a patch into a single token, which hampers reasoning across independent channels and makes them fragile when only a subset of channels is available at test time, a common situation in microscopy and satellite imaging.
Approach
Create one patch token per channel (shared projection and positional embeddings plus a learnable channel embedding), so channel count becomes sequence length; pair with HCS, which first samples the number of channels uniformly and then a random subset of that size during training.
Key results
- On ImageNet, ViT-S/16 with HCS raises red-only validation accuracy from 29.39 to 68.86, and ChannelViT-S/16 reaches 74.64 on RGB versus 71.49 for ViT-S/16 without HCS.
- On JUMP-CP with all 8 channels, ChannelViT-S/16 reaches 68.09% vs 56.87% for ViT-S/16, and ChannelViT-S/8 reaches 74.77% vs 66.44% for ViT-S/8.
- On So2Sat, ChannelViT improves over ViT by 1.28 on the random split (59.75 vs 50.62) and 0.53 on the city split (47.39 vs 41.07).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add HCS during training (ablation) | ViT-S/16 · ImageNet val, R-only | accuracy: 29.39 to 68.86 | Helped |
| use ChannelViT instead of ViT | ImageNet-S/16 with HCS · ImageNet val, RGB | accuracy: 73.01 -> 74.64 | Helped |
| use ChannelViT instead of ViT | JUMP-CP models trained on all 8 channels with HCS · JUMP-CP test, 8 channels | accuracy: 56.87 vs 68.09; 66.44 vs. 74.77 | Helped |
| add HCS during training (ablation) | ViT-S/16 · JUMP-CP test across channel combinations | accuracy: 2.46 -> 35.78 | Helped |
| use ChannelViT instead of ViT | JUMP-CP models trained on 5 fluorescence channels with HCS · JUMP-CP test, 4 channels | accuracy: 43.59 -> 45.94 | Helped |
| use ChannelViT instead of ViT | So2Sat-S/8 with HCS · So2Sat random split | accuracy: +1.28; 50.62 vs 59.75 | Helped |
| use ChannelViT instead of ViT | So2Sat-S/8 with HCS · So2Sat city split | accuracy: +0.53; 41.07 vs 47.39 | Helped |
| tie linear projection weights across channels (ablation) | ChannelViT-S/16 · JUMP-CP test, 5 fluorescence channels | accuracy: 54.78 -> 56.78 | Helped |
| use 8x8 patches instead of 16x16 patches (ablation) | ViT-S with HCS, trained on all JUMP-CP channels · JUMP-CP test, 8 channels | accuracy: 56.87 -> 66.44 | Helped |
| add the 3 brightfield channels to training (ablation) | ViT-S/16 with HCS · JUMP-CP test, 8 channels | accuracy: 55.51 -> 56.87 | Helped |
| increase the share of 8-channel training data (ablation) | ViT-S/16 with HCS · JUMP-CP test, 5 fluorescence channels | accuracy: 55.51 to 45.75 | Hurt |
| increase the share of 8-channel training data (ablation) | ChannelViT-S/16 with HCS · JUMP-CP test, 5 fluorescence channels | accuracy: 56.78 to 57.60 | Helped |
Surprising findings
- Input channel dropout with ViT trained on all channels leads to strong sensitivity to channel combination, and ResNets trained with HCS became unstable (converging to about 5% accuracy).
- MultiViT (one ViT per channel) with HCS dropped sharply on the full 8-channel input (49.06 to 30.25 without vs with HCS context), indicating late channel aggregation hurts.
Limitations
- Sequence length grows with channel count, increasing attention cost (about 1.6x inference time vs ViT).
- Uses standard quadratic attention; efficient attention variants not explored.
- Gains on ImageNet RGB are small because RGB channels are highly correlated.
Experimental setup
Datasets: ImageNet JUMP-CP So2Sat
Benchmarks: ImageNet JUMP-CP So2Sat
Metrics: validation accuracy test accuracy
Baselines: ViT-S/16 ViT-S/8 MultiViT Fully Attentional Networks (FAN) ResNet-50 ResNet-152 input channel dropout
Builds on: Vision Transformer (ViT) channel dropout DINO ViT implementation AdamW
Closest papers by meaning
- Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers (NeurIPS 2024 · similarity 0.91)
- DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images (NeurIPS 2026 · similarity 0.89)
- ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images (CVPR 2024 · similarity 0.88)
- ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning (NeurIPS 2025 · similarity 0.87)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.87)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.