Research map / Architectures and efficiency / CNN design
AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AsCAN, a hybrid convolution-transformer architecture with an asymmetric macro design (more conv blocks early, more transformer blocks late), evaluated on ImageNet classification, ADE20K segmentation, class-conditional and text-to-image diffusion generation.
Problem
Hybrid convolution-transformer architectures are typically designed at the block level with efficient attention approximations, while the macro arrangement of convolutional and transformer blocks across stages is under-explored, leaving latency-performance trade-offs suboptimal across recognition an
Approach
Use vanilla attention and FusedMBConv blocks but distribute them asymmetrically across stages: first stage only convolutions, early stages conv-heavy, later stages transformer-heavy. This macro design gives better throughput than efficient-attention hybrids without optimizing attention kernels, and the same architecture is scaled to a UNet for text-to-image diffusion with a multi-stage training pi
Key results
- AsCAN-T reaches 83.44% top-1 on ImageNet-1K at 3224 images/s on A100 (B=16), versus FasterViT-1 at 83.2% and 1123 images/s.
- On GenEval, the T2I model achieves overall 0.64, above SDXL (0.55) and PixArt-Sigma (0.53).
- On class-conditional ImageNet 256x256, the model reaches FID 2.41 with 52 GFLOPs versus DiT-XL/2-G FID 2.27 at 118 GFLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an asymmetric rather than symmetric distribution of convolution and transformer blocks (ablation) | AsCAN image classifier · ImageNet-1K | top-1 accuracy: 82.8% -> 83.4% | Helped |
| Put convolution blocks before transformer blocks within stages (ablation) | AsCAN image classifier · ImageNet-1K | throughput, A100 B=16 (images/s): 1428 -> 3224 | Helped |
| Keep convolution blocks in the stem instead of transformer blocks (ablation) | AsCAN image classifier · ImageNet-1K | throughput, A100 B=64 (images/s): 1584 -> 4295 | Helped |
| Replace MBConv with FusedMBConv (ablation) | AsCAN hybrid block · ImageNet-1K | throughput, A100 B=64 (images/s): 3013 -> 4295 | Helped |
| Use vanilla attention instead of multi-axial attention (ablation) | AsCAN transformer block · ImageNet-1K | throughput, A100 B=64 (images/s): 3541 -> 4295 | Helped |
| Increase transformer blocks in the network from configuration C6 to C9 (ablation) | AsCAN image classifier · ImageNet-1K | top-1 accuracy: 82.8% -> 82.7% | No clear effect |
| Use sampled cfg (ablation) | AsCAN class-conditional UNet · ImageNet-1K 256×256 class-conditional generation | FID: 2.41 -> 2.23 | Helped |
| Use AsCAN rather than FasterViT-1 | ImageNet-1K classification · ImageNet-1K | throughput, A100 B=16 (images/s): 1123 -> 3224 | Helped |
| Use AsCAN rather than SDXL | Text-to-image generation · GenEval | overall score: 0.55 -> 0.64 | Helped |
| Use AsCAN rather than SDXL | Text-to-image generation · Validation Set-B-10K | FID: 35.86 -> 15.45 | Helped |
| Use AsCAN-T rather than Swin-T as the UPerNet backbone | UPerNet · ADE20K | latency, A100 (frames/s): 44 -> 64 | Helped |
Surprising findings
- Using vanilla quadratic attention with an asymmetric macro design gives higher throughput than many efficient-attention hybrids, even without attention kernel optimization.
- Placing transformer blocks in early stages lowers throughput substantially, and T-before-C orderings yield worse accuracy-throughput trade-offs.
Limitations
- Uses vanilla quadratic attention without efficient attention alternatives, so cost grows quadratically with resolution.
- Large-scale T2I training data is proprietary/licensed and not released, limiting reproducibility.
- Text-to-image comparisons with baselines are partly qualitative and based on a user study of 1000 prompts.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K ADE20K GenEval proprietary/licensed T2I image-text data
Benchmarks: ImageNet-1K ImageNet-21K ADE20K GenEval ImageNet-1K class-conditional FID
Metrics: top-1 accuracy throughput (images/s) FID mIoU GenEval overall score user preference rate
Baselines: MaxViT CoAtNet FasterViT PixArt-alpha PixArt-Sigma SDXL DiT-XL/2-G U-ViT ADM LDM Swin SMT
Builds on: CoAtNet MaxViT ConvNeXt PixArt-alpha DDPM
Closest papers by meaning
- Bottleneck Transformers for Visual Recognition (CVPR 2021 · similarity 0.88)
- Involution: Inverting the Inherence of Convolution for Visual Recognition (CVPR 2021 · similarity 0.88)
- CMT: Convolutional Neural Networks Meet Vision Transformers (CVPR 2022 · similarity 0.88)
- Contextual Convolutional Networks (ICLR 2023 · similarity 0.87)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.87)
- LSNet: See Large, Focus Small (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.