Research map / Vision-language and multimodal / Human and embodied modeling
Locality Alignment Improves Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes locality alignment, a self-supervised post-training stage (MaskEmbed) that fine-tunes pre-trained ViTs via masked embedding reconstruction of a frozen teacher, improving patch-level semantics and downstream VLM spatial understanding.
Problem
Pre-trained ViTs trained with image-level supervision (e.g., CLIP, SigLIP) encode global semantics but poorly encode per-region semantics, which limits spatial reasoning in VLMs that use them as frozen or partially trained backbones.
Approach
Use a frozen pre-trained ViT as a teacher on masked views and fine-tune the ViT (encoder) with a transformer decoder to reconstruct the teacher's masked embedding sequence from masked patch embeddings, yielding patch embeddings that encode local semantics without new annotations; the decoder is then reused as the VLM adapter.
Key results
- Locality alignment improves CLIP ViT-L @ 336px local probing from 46.05 to 52.71 and global from 55.13 to 57.75 on MSCOCO.
- Locality alignment improves SigLIP SO400M @ 384px local probing from 50.25 to 53.00.
- CLIPSelf-style objective degrades CLIP ViT-B probing (local 36.16 vs teacher 44.63), while MaskEmbed improves it (local 46.32, global 54.55 with 3 masks).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply locality alignment | CLIP ViT-L @ 336px · MSCOCO val local probing | macro-averaged recall: 46.05 -> 52.71 | Helped |
| Apply locality alignment | SigLIP SO400M @ 384px · MSCOCO val local probing | macro-averaged recall: 50.25 -> 53.00 | Helped |
| Apply locality alignment | CLIP ViT-B · MSCOCO val global probing | macro-averaged recall: 52.61 -> 54.55 | Helped |
| Apply locality alignment | MAE ViT-B · MSCOCO val local probing | macro-averaged recall: 39.46 -> 37.80 | Hurt |
| Apply locality alignment | DINOv2 ViT-B · MSCOCO val global probing | macro-averaged recall: 56.95 -> 55.64 | Hurt |
| Use CLIPSelf crop-based reconstruction | CLIP ViT-B · MSCOCO val local probing | macro-averaged recall: 44.63 -> 36.16 | Hurt |
| Use averaged features instead of a transformer decoder (ablation) | MaskEmbed on CLIP ViT-B · MSCOCO val local probing | macro-averaged recall: 46.32 -> 40.97 | Hurt |
| Use the [CLS] token rather than the second-to-last-layer embedding sequence as reconstruction target (ablation) | MaskEmbed on IN1k ViT-B/16 · MSCOCO val local probing | macro-averaged recall: 45.66 -> 44.16 | Hurt |
| Add Mixup augmentation (ablation) | MaskEmbed on IN1k ViT-B/16 · MSCOCO val local probing | macro-averaged recall: 45.66 -> 45.72 | No clear effect |
| Add CutMix augmentation (ablation) | MaskEmbed on IN1k ViT-B/16 · MSCOCO val global probing | macro-averaged recall: 51.43 -> 48.60 | Hurt |
| Add the null mask to uniform mask sampling (ablation) | MaskEmbed on IN1k ViT-B/16 · MSCOCO val global probing | macro-averaged recall: 49.17 -> 51.43 | Helped |
| Use MSE reconstruction loss instead of masked-only MSE (ablation) | MaskEmbed on IN1k ViT-B/16 · MSCOCO val local probing | macro-averaged recall: 42.48 -> 45.66 | Helped |
Surprising findings
- CLIPSelf-style reconstruction with average-pooled features degrades CLIP probing performance, suggesting the weak decoder rather than cropping is the issue.
- Using an MLP adapter on locality-aligned features slightly hurts VLM performance; the trained MaskEmbed decoder must be used as the adapter.
Limitations
- Only evaluated with a single VLM training approach (Llava-style, frozen ViT, Prismatic recipe) and Llama-2 7B LM.
- Patch embeddings are less interpretable because they lie in a different space than the teacher's outputs.
- Does not preserve CLIP's zero-shot classification abilities.
Experimental setup
Datasets: ImageNet-1k ImageNet-21k MSCOCO Llava-1.5 mixture LVIS-Instruct-4V LRV-Instruct
Benchmarks: MSCOCO patch-level probing RefCOCO OCID-Ref TallyQA VSR AI2D TextVQA VQAv2 VizWiz POPE ImageNet-1k classification
Metrics: macro-averaged recall accuracy
Baselines: CLIPSelf CLIP ViT-L @ 336px SigLIP SO400M @ 384px DINOv2 feature fusion LocCa
Builds on: CLIPSelf Prismatic VLM Llava-1.5 BEiT
Closest papers by meaning
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.91)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.91)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
- RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · similarity 0.90)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.