Research map / Vision-language and multimodal / CLIP and zero-shot

Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling

NeurIPS 2026 · Vision-language and multimodal › CLIP and zero-shot · approach: Fine-grained improvements to CLIP

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SALM, an unsupervised framework that aligns CLIP's patch features with a frozen DINOv2 reference via dual-matrix alignment and latent masked modeling (plus a self-distillation variant, SALM-Self), trained only on ImageNet-1K images, to improve fine-grained perception while preserving zero-shot ability.

Problem

CLIP captures global semantics but lacks fine-grained local perception, limiting dense prediction and MLLM visual understanding; prior enhancement methods (multi-expert, distillation, generative, kernel alignment) either add cost, disrupt CLIP's embedding space, or neglect intra-sample spatial struc

Approach

Explicitly align intra-image patch relations (cosine-similarity Spatial Relation Matrix and z-scored norm-difference Energy Difference Matrix) with a vision-centric teacher, and implicitly align by reconstructing masked teacher patch latents through a lightweight Cross-Guided Adapter conditioned on CLIP features; a self-distillation variant uses CLIP's own shallow layer as reference.

Key results

• SALM reaches 69.25% average zero-shot accuracy over 11 datasets vs 67.35% for CLIP, with ImageNet-1K 77.21% vs 76.55%.
• SALM reaches 42.83 mIoU on ADE20K linear probing vs 36.96 for CLIP and 36.38 for KUEA.
• In LLaVA-1.5-7B integration, SALM averages 66.69 vs 60.82 for the LLaVA baseline and 64.92 for KUEA.

Surprising findings

• Simple feature distillation drops SVHN accuracy from 55.97% to 51.87%, showing that rigid point-wise feature matching disrupts CLIP's semantic space.
• RADIOv2.5 reproduction degrades zero-shot average to 58.87% from 67.35%.

Limitations

• Open-vocabulary segmentation remains far below DeCLIP, since SALM uses no patch-level text supervision.
• Performance depends on feature-map size mismatch handled by interpolation between source and reference models.

Experimental setup

Datasets: ImageNet-1K

Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 ADE20K VOC2012 COCO-Stuff Cityscapes RefCOCO VQA v2 POPE

Metrics: top-1 accuracy mIoU

Baselines: RADIOv2.5 un$^2$CLIP KUEA DeCLIP simple feature distillation

Builds on: CLIP DINOv2 KUEA masked image modeling LLaVA

Closest papers by meaning

• Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.91)
• VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.91)
• Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.90)
• Refining CLIP's Spatial Awareness: A Visual-Centric Perspective (ICLR 2025 · similarity 0.90)
• Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)