AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks

Tingyu Qu, Tinne Tuytelaars, Marie-Francine Moens

ECCV 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 4 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add routing functions to LoRA/Adapter low-rank bottleneckLoRA/Adapter with RoBERTa-base + ViT-B/16 · VQAv2accuracy: 44.15 -> 54.21 (LoRA r=64, x_t x''_v)Improved
add routing functions to LoRA with GPT2LoRA with GPT2-base + ViT-B/16 · COCO Cap. (r=64)CIDEr: 55.8 -> 88.7Improved
increase bottleneck rank from 64 to 128 (ablation)LoRA/Adapter with routing on VQAv2 · VQAv2accuracy: 53.88 -> 53.09 (LoRA x_t(x_v)^T x_v)No clear effect
add routing function to LoRA when reducing rank 128 to 64 (ablation)LoRA on COCO Cap. · COCO Cap.CIDEr: 70.3 -> 55.8 without routing; routing stays at 88.7 / 87.9Improved
use routing x_t (x_v)^T x_v instead of cross-attention in LoRA (ablation)LoRA on COCO Cap. · COCO Cap. (cross-attention comparison)CIDEr: 92.2 -> 99.4Improved
use routing x_t (x_v)^T x_v instead of cross-attention in Adapter (ablation)Adapter on COCO Cap. · COCO Cap. (cross-attention comparison)CIDEr: 99.8 -> 98.8No clear effect
add ReLU nonlinearities inside the routing function (ablation)x_t (x_v)^T x_v routing · VQAv2 and COCO Cap. (r=64)accuracy / CIDEr: 53.88 -> 53.45 VQA (LoRA)No clear effect
replace visual CLS features with random noise as x_R (ablation)LoRA + x_t (x_v)^T x_v · VQAv2 (r=64)accuracy: 53.88 -> 43.80Hurt
replace visual CLS features with all-ones as x_R (ablation)Adapter + x_t (x_v)^T x_v · VQAv2 (r=64)accuracy: 54.48 -> 24.42Hurt
add routing functions (inference cost)LoRA/Adapter · VQAv2 inference time (ms)inference time: 0.2375 -> 0.2533 (LoRA, x_t o x'_v)No clear effect
use larger backbones (ViT-L/16 + RoBERTa-large) (ablation)Adapter with routing on VQAv2 · VQAv2 (r=64)accuracy: 44.82 -> 55.43 (x_t (x_v)^T x_v)Improved
add routing functions to CLIP-BART with single LoRACLIP-BART single LoRA · CLIP-BART avg of VQA/GQA/NLVR2accuracy: 63.80 -> 64.35Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy