Research map / Vision-language and multimodal / CLIP and zero-shot

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering

ICML 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Test-time adaptation of VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Double-Filter slims fine-tuning of vision-language pre-trained models by filtering image patches (YOLO foreground/background split plus ViT [CLS] attention) and genetically searching which fine-grained sub-layers to replace with adapters.

Problem

Fine-tuning vision-language pre-trained models is computationally expensive, and existing efficient fine-tuning methods either add adapter overhead or remove whole transformer blocks coarsely while ignoring image-patch redundancy.

Approach

Combine a data-level Image Patch Filter that keeps high-[CLS]-attention patches separately within YOLO-detected foreground and background regions with an adjustable genetic algorithm that selects which fine-grained sub-layers (MHSA, MHCA, FFN) to replace with lightweight adapters.

Key results

• On METER, Double-Filter reduces FLOPs by 21.18G on VQA and 12.51G on NLVR versus DAS's 11.16G and 5.13G, with VQA test-dev 74.25 vs DAS 74.80.
• On METER NLVR2 test-P, Double-Filter scores 80.12 against 80.11 for DAS and 80.38 for Scaled PA, with the average drop around 0.30% relative to full tuning.
• Inference throughput on METER VQA rises from 55.32 items/s (full tuning) to 70.72 items/s with Double-Filter.

Measured effects reported in the paper

Surprising findings

• Filtering some image patches on NLVR2 sometimes improved METER accuracy over the unfiltered baseline, suggesting redundant patches can hurt performance.
• ViLT loses noticeable accuracy after removing only 4 layers, while METER tolerates removing 24 of 36 layers on NLVR.

Limitations

• Reliance on ViT attention for patch importance and on a pre-trained YOLO detector may limit flexibility.
• The genetic algorithm has high computational complexity and may converge to local optima.
• Retrieval results show slightly lower performance than some PEFT baselines on METER.

Experimental setup

Datasets: VQA2.0 NLVR2 Flickr30K

Benchmarks: VQA2.0 NLVR2 Flickr30K

Metrics: accuracy Recall@1 FLOPs inference throughput (items/s)

Baselines: Full Tuning Classifier Only Shallow Prompt Deep Prompt LoRA Adapter Scaled PA DAS EViT

Builds on: DAS EViT Adapter METER ViLT

Closest papers by meaning

• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.88)
• Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.88)
• Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.88)
• Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.88)
• SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes (NeurIPS 2025 · similarity 0.88)
• Vision-Language Models are Strong Noisy Label Detectors (NeurIPS 2024 · similarity 0.87)