Research map / Vision-language and multimodal / CLIP and zero-shot
p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes p-adapter, a PETL adapter placed after attention that re-normalizes attention weights with p-Laplacian message passing (learnable p) to handle heterophilic attention graphs in generative VLMs.
Problem
Standard adapters after attention in pre-trained VLMs act as low-pass graph convolutions on heterophilic attention graphs (query and value features lie in distinct spaces), limiting their effectiveness for parameter-efficient transfer.
Approach
Recast adapter-after-attention as one-layer GCN message passing on a bipartite attention graph, then replace it with p-Laplacian message passing whose attention weights are re-normalized using node features and a layer-wise learnable p, enabling dynamic low/high-frequency filtering.
Key results
- p-Adapter with BLIP CapFilt-L tunes 6.39% of parameters and reaches 70.39% on VQA2.0 vs 69.53% for vanilla adapter and 70.56% for full fine-tuning.
- With mPLUG ViT-B, p-Adapter averages 76.01 across six benchmarks, exceeding full fine-tuning (75.76) and vanilla adapter (74.73).
- On SNLI-VE, p-Adapter improves over full fine-tuning by 1.05% with BLIP and 0.54% with mPLUG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use p-Adapter instead of Adapter | BLIP CapFilt-L · VizWizVQA test-dev | accuracy: 35.37 -> 37.16 | Helped |
| Use p-Adapter instead of Adapter | mPLUG ViT-B · SNLI-VE test-P | accuracy: 78.56 -> 79.26 | Helped |
| Use p-Adapter instead of Adapter | BLIP CapFilt-L · COCO Captions Karpathy test | CIDEr: 128.8 -> 130.9 | Helped |
| Replace GCN message passing with APPNP (ablation) | BLIP CapFilt-L p-Adapter · VQA2.0 | accuracy: 69.53 -> 70.22 | Helped |
| Replace GCN message passing with GCNII (ablation) | BLIP CapFilt-L p-Adapter · VQA2.0 | accuracy: 69.53 -> 70.13 | Helped |
| Use p-GNN message passing instead of GCN (ablation) | BLIP CapFilt-L · VQA2.0 | accuracy: 69.53 -> 70.39 | Helped |
| Replace query features with zeros in augmented attention (ablation) | BLIP CapFilt-L p-Adapter · VQA2.0 | accuracy: 70.39 -> 70.02 | Hurt |
| Replace query features with random noise in augmented attention (ablation) | BLIP CapFilt-L p-Adapter · VQA2.0 | accuracy: 70.39 -> 69.90 | Hurt |
| Use fixed p=1.25 instead of learnable p (ablation) | BLIP CapFilt-L p-Adapter · SNLI-VE | accuracy: 79.40 -> 78.84 | Hurt |
| Use fixed p=1.50 instead of learnable p (ablation) | BLIP CapFilt-L p-Adapter · SNLI-VE | accuracy: 79.40 -> 78.90 | Hurt |
| Use fixed p=1.75 instead of learnable p (ablation) | BLIP CapFilt-L p-Adapter · SNLI-VE | accuracy: 79.40 -> 78.94 | Hurt |
| Add p-adapters after self-attention, retaining FFN insertion (ablation) | BLIP CapFilt-L p-Adapter · VQA2.0 | accuracy: +0.90 | Helped |
Surprising findings
- Fixed-p variants (p=1.25, 1.5, 1.75) gain only modestly over vanilla adapter, while the layer-wise learnable p gives larger gains, suggesting the best p varies per task and layer.
Limitations
- Experiments limited to two generative VLMs (BLIP CapFilt-L and mPLUG ViT-B) and three task families.
- Encoder is frozen and only the decoder is trained for new tasks, a restricted adaptation setting.
- Ablations run only on BLIP with VQA2.0, SNLI-VE and COCO Captions due to compute budget.
Experimental setup
Datasets: VQA2.0 VizWizVQA SNLI-VE COCO Captions TextCaps VizWizCaps
Benchmarks: VQA2.0 VizWizVQA SNLI-VE COCO Captions TextCaps VizWizCaps
Metrics: accuracy BLEU@4 CIDEr
Baselines: full fine-tuning adapter prefix tuning LoRA APPNP GCNII
Builds on: adapter tuning p-Laplacian message passing (p-GNN) GCN BLIP mPLUG
Closest papers by meaning
- Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model (ICLR 2025 · similarity 0.89)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.89)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.88)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.88)
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.87)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.