Research map / Vision-language and multimodal / CLIP and zero-shot
From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ABS, a training-free method that uses DINO attention to guide cropping in both raw image and CLIP feature space, plus soft matching of LLM descriptions, improving zero-shot CLIP classification and OOD generalization.
Problem
Random cropping used to align CLIP with LLM-generated fine-grained descriptions can crop background objects and cause the model to overfocus on local details, losing global semantic context.
Approach
Guide crop centers with DINO's last-layer attention map (top-k sampling), perform the same crops on intermediate CLIP feature maps so crops retain global category information, and softly weight LLM descriptions per crop via a softmax over similarities.
Key results
- ABS with ViT-B/16 reaches 66.00% average OOD accuracy vs 64.03% for WCA, with up to +6.21 points on ImageNet-A (ViT-B/32).
- On the zero-shot benchmark ABS achieves the best results on five of six datasets, e.g., 67.74% ImageNet with ViT-B/32 vs 66.84% WCA.
- Ablation on ViT-B/16 shows full ABS improves average over CuPL by +2.98 points, versus +0.33 for raw-space selection alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace random cropping with DINO-B/16 attention-guided cropping (ablation) | ABS · ImageNet average across ImageNet, CUB, DTD, and ImageNet-A | top-1 accuracy (%): 57.47 -> 62.25 | Helped |
| Add attention-guided raw-space selection to CuPL (ablation) | CuPL / CLIP ViT-B/16 · ImageNet | top-1 accuracy (%): 69.61 -> 69.34 | Hurt |
| Add attention-guided raw-space selection to CuPL (ablation) | CuPL / CLIP ViT-B/16 · DTD | top-1 accuracy (%): 50.53 -> 51.52 | Helped |
| Add attention-guided feature-space selection to CuPL (ablation) | CuPL / CLIP ViT-B/16 · ImageNet-V2 | top-1 accuracy (%): 63.27 -> 63.10 | No clear effect |
| Add soft matching to CuPL (ablation) | CuPL / CLIP ViT-B/16 · DTD | top-1 accuracy (%): 50.53 -> 52.89 | Helped |
| Add both raw-space and feature-space selection to CuPL (ablation) | CuPL / CLIP ViT-B/16 · ImageNet | top-1 accuracy (%): 69.61 -> 70.32 | Helped |
| Add raw-space selection, feature-space selection, and soft matching to CuPL (ablation) | CuPL / CLIP ViT-B/16 · ImageNet | top-1 accuracy (%): +2.98 | Helped |
| Use crop augmentation instead of mask in raw space (ablation) | ABS / CLIP ViT-B/16 · ImageNet | top-1 accuracy (%): 53.65 -> 70.98 | Helped |
| Use crop augmentation instead of mask in feature space (ablation) | ABS / CLIP ViT-B/16 · ImageNet | top-1 accuracy (%): 62.72 -> 70.31 | Helped |
| Increase feature-selection layer depth from Layer_id 1 to Layer_id 11 (ablation) | ABS / CLIP · ImageNet | accuracy (%): 66.14 -> 67.74 | Helped |
Surprising findings
- Smaller crop ratios can outperform larger ones once feature-space selection is included, contrary to WCA's trend.
- Mask-based augmentation performs poorly in both raw and feature space, while crop is best, attributed to CLIP's pretraining using crops.
Limitations
- Relies on the attention map of a single model (DINO); exploring SAM is future work.
- Crop-based augmentation is the only augmentation found effective; other augmentations are not explored.
- Limited to image-text modalities; slower than CLIP baseline at inference.
Experimental setup
Datasets: ImageNet ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-A CUB Oxford Pets DTD Food101 Place365
Benchmarks: ImageNet ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-A CUB Oxford Pets DTD Food101 Place365
Metrics: top-1 accuracy
Baselines: CLIP CLIP-E CLIP-D CuPL Waffle WCA CoOp CoCoOp UPT ProGrad KgCoOp TPT
Builds on: WCA DINO CuPL CLIP
Closest papers by meaning
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling (NeurIPS 2026 · similarity 0.89)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.89)
- Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.