Research map / Vision-language and multimodal / Human and embodied modeling
Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAPS, a linguistic-aware visual patch selection and calibration framework that slims ViT patches for patch-word alignment in image-text retrieval without object detectors.
Problem
Fine-grained cross-modal alignment normally relies on pre-trained object detectors for region features, which is costly and error-prone; directly using ViT patches for patch-word alignment suffers from patch redundancy and semantic ambiguity.
Approach
Uses language-context scores (cross-attention to text plus image self-attention) with Gumbel-Softmax differentiable patch selection to drop redundant patches, then aggregates significant patches into semantically complete patches and fuses redundant ones, followed by sparse bidirectional max-alignment patch-word matching.
Key results
- On Flickr30K 1K with ViT-Base-384 + BERT-base, LAPS reaches 525.4 rSum versus 515.8 for SGR and 512.4 for CHAN.
- On MS-COCO 5K with Swin-Base-384 + BERT-base, LAPS reaches 470.1 rSum versus 462.2 for VSE++ (best baseline on that setting).
- On zero-shot RefCOCO val with CLIP ViT-B/16 backbone, LAPS reaches 44.2 versus 41.8 for SCAN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply LAPS instead of VSE++ | ViT-Base-224 + BERT-base image-text retrieval · Flickr30K 1K | rSum: 496.1 -> 507.3 | Helped |
| apply LAPS instead of CHAN | ViT-Base-224 + BERT-base image-text retrieval · MS-COCO 1K | rSum: 522.2 -> 526.3 | Helped |
| apply LAPS instead of CHAN | ViT-Base-224 + BERT-base image-text retrieval · MS-COCO 5K | rSum: 428.0 -> 434.4 | Helped |
| add patch selection (ablation) | LAPS · Flickr30K 1K | R@1: 69.2 -> 74.0 | Helped |
| add language-context supervision to patch selection (ablation) | LAPS · Flickr30K 1K | R@1: 71.1 -> 74.0 | Helped |
| use predictive scores in addition to attentive scores for patch significance (ablation) | LAPS · Flickr30K 1K | R@1: 73.5 -> 74.0 | Helped |
| add patch calibration (ablation) | LAPS · Flickr30K 1K | R@1: 70.4 -> 74.0 | Helped |
| add fusion of redundant patches (ablation) | LAPS · Flickr30K 1K | R@1: 73.5 -> 74.0 | Helped |
| replace learned patch aggregation with clustering (ablation) | LAPS · Flickr30K 1K | R@1: 74.0 -> 68.4 | Hurt |
| replace sparse patch-word alignment with SCAN (ablation) | LAPS · Flickr30K 1K | R@1: 74.0 -> 71.3 | Hurt |
| remove word-to-patch alignment (ablation) | LAPS · Flickr30K 1K | R@1: 74.0 -> 70.9 | Hurt |
| remove patch-to-word alignment (ablation) | LAPS · Flickr30K 1K | R@1: 74.0 -> 72.7 | Hurt |
Surprising findings
- Existing fine-grained region-based methods (SCAN, SGR, CHAN) underperform the coarse-grained VSE++ when applied to ViT patch features, especially at higher patch counts.
- Word slimming (selecting or aggregating textual words) hurts performance, which the authors attribute to high information density of text.
Limitations
- Word slimming does not help because text lengths in Flickr30K and MS-COCO are short, so the framework's benefit on text is limited.
- Clustering-based aggregation (DPC-KNN) lowers performance and efficiency compared to the proposed aggregation.
- Performance is sensitive to selection and aggregation ratios, with small ratios hurting results especially for Swin encoders.
Experimental setup
Datasets: Flickr30K MS-COCO
Benchmarks: Flickr30K MS-COCO RefCOCO RefCOCO+ RefCOCOg
Metrics: R@1 R@5 R@10 rSum
Baselines: VSE++ SCAN SGR CHAN HREM TGDT UNITER VILT SOHO ALBEF BLIP CLIP
Builds on: SCAN VSE++ SGR CHAN Gumbel-Softmax ViT
Closest papers by meaning
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.89)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.88)
- MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding (ICLR 2026 · similarity 0.88)
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.