Research map / Vision-language and multimodal / Human and embodied modeling

ResCLIP: Residual Attention for Training-free Dense Vision-language Inference

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Language-grounded scene text and structure · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ResCLIP, a training-free plug-and-play method that remolds CLIP's last-block attention using cross-correlation attention from intermediate layers (RCS) plus semantic-segmentation-guided refinement (SFR), improving open-vocabulary semantic segmentation.

Problem

CLIP's final-layer self-attention is spatially invariant, so its dense vision-language predictions are poor for open-vocabulary semantic segmentation; existing training-free fixes use only self-correlation attention and ignore cross-correlation (query-key) attention from intermediate layers.

Approach

Observes that query-key cross-correlation attention in non-final CLIP layers is class-specific and localized; averages it across layers and residually mixes it into the last-block self-correlation attention (RCS), then refines the attention with a segmentation-mask-based decay and Gaussian smoothing (SFR).

Key results

• Integrated with NACLIP (ViT-B/16), ResCLIP reaches 43.2% mIoU on datasets with background class (+1.8) and 40.3 average on datasets without background (+2.1).
• With SCLIP on ViT-L/14, ResCLIP improves average mIoU from 23.6 to 36.7 (+13.1) on datasets without background class.
• On VOC20 ablation with NACLIP, RCS alone gives 85.5 mIoU (+5.8) and SFR alone +1.8; combined 86.0 (+6.3).

Measured effects reported in the paper

Surprising findings

• Intermediate-layer query-key attention, which is typically spatially invariant in the last layer, carries class-specific localization properties usable for dense prediction.

Limitations

• Hyperparameters lambda_rcs and lambda_sfr require tuning, though performance is reported as stable over a range.
• Ablations are run only on NACLIP with VOC20; SFR depends on an initial segmentation mask from the model itself.

Experimental setup

Benchmarks: PASCAL VOC20 PASCAL Context59 COCO-Stuff Cityscapes ADE20K PASCAL VOC21 PASCAL Context60 COCO Object

Metrics: mIoU

Baselines: CLIP MaskCLIP ReCo CLIPSurgery GEM SCLIP NACLIP ClearCLIP GroupViT TCL

Builds on: CLIP SCLIP NACLIP ClearCLIP

Closest papers by meaning

• Refining CLIP's Spatial Awareness: A Visual-Centric Perspective (ICLR 2025 · similarity 0.89)
• Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.88)
• Grounding Everything: Emerging Localization Properties in Vision-Language Transformers (CVPR 2024 · similarity 0.88)
• DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception (CVPR 2025 · similarity 0.88)
• CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large Vision-Language Models (ICML 2026 · similarity 0.88)
• un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.88)