Research map / Architectures and efficiency / Pruning and efficient tuning
Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks
Tingyu Qu, Tinne Tuytelaars, Marie-Francine Moens
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add routing functions to LoRA/Adapter low-rank bottleneck | LoRA/Adapter with RoBERTa-base + ViT-B/16 · VQAv2 | accuracy: 44.15 -> 54.21 (LoRA r=64, x_t x''_v) | Improved |
| add routing functions to LoRA with GPT2 | LoRA with GPT2-base + ViT-B/16 · COCO Cap. (r=64) | CIDEr: 55.8 -> 88.7 | Improved |
| increase bottleneck rank from 64 to 128 (ablation) | LoRA/Adapter with routing on VQAv2 · VQAv2 | accuracy: 53.88 -> 53.09 (LoRA x_t(x_v)^T x_v) | No clear effect |
| add routing function to LoRA when reducing rank 128 to 64 (ablation) | LoRA on COCO Cap. · COCO Cap. | CIDEr: 70.3 -> 55.8 without routing; routing stays at 88.7 / 87.9 | Improved |
| use routing x_t (x_v)^T x_v instead of cross-attention in LoRA (ablation) | LoRA on COCO Cap. · COCO Cap. (cross-attention comparison) | CIDEr: 92.2 -> 99.4 | Improved |
| use routing x_t (x_v)^T x_v instead of cross-attention in Adapter (ablation) | Adapter on COCO Cap. · COCO Cap. (cross-attention comparison) | CIDEr: 99.8 -> 98.8 | No clear effect |
| add ReLU nonlinearities inside the routing function (ablation) | x_t (x_v)^T x_v routing · VQAv2 and COCO Cap. (r=64) | accuracy / CIDEr: 53.88 -> 53.45 VQA (LoRA) | No clear effect |
| replace visual CLS features with random noise as x_R (ablation) | LoRA + x_t (x_v)^T x_v · VQAv2 (r=64) | accuracy: 53.88 -> 43.80 | Hurt |
| replace visual CLS features with all-ones as x_R (ablation) | Adapter + x_t (x_v)^T x_v · VQAv2 (r=64) | accuracy: 54.48 -> 24.42 | Hurt |
| add routing functions (inference cost) | LoRA/Adapter · VQAv2 inference time (ms) | inference time: 0.2375 -> 0.2533 (LoRA, x_t o x'_v) | No clear effect |
| use larger backbones (ViT-L/16 + RoBERTa-large) (ablation) | Adapter with routing on VQAv2 · VQAv2 (r=64) | accuracy: 44.82 -> 55.43 (x_t (x_v)^T x_v) | Improved |
| add routing functions to CLIP-BART with single LoRA | CLIP-BART single LoRA · CLIP-BART avg of VQA/GQA/NLVR2 | accuracy: 63.80 -> 64.35 | Improved |
Closest papers by meaning
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.88)
- FATE: Feature-Adapted Parameter Tuning for Vision-Language Models (AAAI 2025 · similarity 0.88)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.87)
- Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.87)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
