Research map / Architectures and efficiency / Attention and transformers
Learned Queries for Efficient Local Attention
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a 7×7 rather than 3×3 QnA window (ablation) | QnA-Tiny · ImageNet-1K | Top-1 Acc.: 81.7 -> 82.0 | Helped |
| Replace DETR transformer blocks with QnA blocks (ablation) | DETR with QnA-Tiny backbone · COCO 2017 detection | AP: 37.9 -> 38.5 | Helped |
| Use a 7×7 rather than 3×3 receptive field (ablation) | DETR with QnA-Tiny backbone · COCO 2017 detection | AP: 37.5 -> 37.9 | Helped |
| Replace baseline downsampling with QnA downsampling (ablation) | ImageNet-1K design-choice model · ImageNet-1K | Top1-Acc.: 81.2 -> 81.9 | Helped |
| Increase the number of learned queries from one to two (ablation) | QnA · ImageNet-1K, lightweight QnA-ViT | Top-1 Acc.: 80.3 -> 80.7 | Helped |
| Increase learned queries from two to four (ablation) | QnA · ImageNet-1K, lightweight QnA-ViT | Top-1 Acc.: 80.7 -> 80.81 | No clear effect |
| Increase QnA heads from [2,2,2,2] to [8,16,32,64] (ablation) | Shallow QnA network · ImageNet-1K, 90 epochs | Top-1 Acc.: 68.33 -> 71.12 | Helped |
| Use more QnA blocks in early stages instead of local self-attention (ablation) | QnA-ViT · ImageNet-1K | Top- Acc: 81.70 -> 81.86 | Helped |
| Use more global-attention blocks and fewer QnA blocks in stage 3 (ablation) | QnA-ViT · ImageNet-1K | Top- Acc: 81.9 -> 80.8 | Hurt |
Closest papers by meaning
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.89)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- Dissecting Query-Key Interaction in Vision Transformers (NeurIPS 2024 · similarity 0.88)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.88)
- LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.88)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.