Research map / Architectures and efficiency / Pruning and efficient tuning
UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniPT, a memory-efficient parameter-efficient transfer learning method that trains a lightweight parallel network with top-down interaction guided by the last-layer output and a confidence-based cross-layer aggregation, avoiding backpropagation through the backbone.
Problem
Parameter-efficient transfer learning methods still incur high training memory because gradients flow through the backbone; existing memory-efficient side-network methods (e.g., LST) lack scalability, adaptability, and generalizability across architectures.
Approach
A parallel network detached from the backbone's intermediate activations, where each layer's features are refined by attention guided by the last-layer output (parallel interaction), and cross-layer features are fused with input-dependent confidence weights (confidence aggregation) instead of LST's static gated ladder.
Key results
- On MSR-VTT with CLIP4Clip, UniPT reaches 361.3 Rsum at 3.4G training memory versus 358.8 for Prompt tuning and 356.7 for LST at 8.0G (T5-free comparison; LST 356.7 at 8.0G).
- On GLUE with T5-base, UniPT reduces training memory from 17.6G (full fine-tuning) to 2.9G with average score 84.3 versus LST 84.1.
- On MSCOCO 5K with VSE∞, UniPT improves over LST by 3.3 R@1 points in I-T direction (absolute R@1 improvement reported in text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LST with UniPT | VSE∞ · Flickr30K | Rsum: 529.5 -> 537.4 | Helped |
| replace LST with UniPT | VSE∞ · MSCOCO5K | Rsum: 434.5 -> 445.3 | Helped |
| replace LST with UniPT | CLIP4Clip · MSR-VTT | Rsum: 356.7 -> 361.3 | Helped |
| replace LST with UniPT | CLIP4Clip · MSVD | Rsum: 407.2 -> 432.1 | Helped |
| replace LST with UniPT | CLIP-ViL · VQAv2 Test-Dev | accuracy: 75.29 -> 75.33 | Helped |
| replace LST with UniPT | CLIP-ViL · GQA Test-Dev | accuracy: 59.93 -> 60.10 | Helped |
| replace LST with UniPT | MDETR · RefCOCO TestA | accuracy: 85.19 -> 86.25 | Helped |
| add last-layer interaction guidance (ablation) | UniPT · Flickr30K | Rsum | Helped |
| replace average pooling or static ladder gating with confidence aggregation (ablation) | UniPT · Flickr30K | Rsum | Helped |
| replace pre-interaction pooling with truncated attention (ablation) | UniPT CNN · Flickr30K | Rsum: 517.8 -> 537.4 | Helped |
| replace pre-interaction multi-head self-attention with truncated attention (ablation) | UniPT CNN · Flickr30K | Rsum: 525.3 -> 537.4 | Helped |
| replace post-interaction multi-head self-attention with truncated attention (ablation) | UniPT CNN · Flickr30K | Rsum: 494.8 -> 537.4 | Helped |
Surprising findings
- More powerful guidance backbones (ViT-L/14 as guidance for ViT-B/32) bring further gains, even though the guidance model does not directly contribute to the output.
Limitations
- Performance gap remains relative to full fine-tuning.
- Large input sizes such as shallow CNN feature maps increase computational complexity.
- The pre-post interaction design reduces memory but increases trainable parameters somewhat.
Experimental setup
Datasets: MSCOCO Flickr30K MSR-VTT MSVD VQAv2 GQA RefCOCO RefCOCO+ RefCOCOg GLUE
Benchmarks: Flickr30K MSCOCO MSR-VTT MSVD VQAv2 GQA RefCOCO RefCOCO+ RefCOCOg GLUE
Metrics: Recall@1 Rsum accuracy F1 Matthews correlation Pearson-Spearman correlation training memory (GB)
Baselines: Fully fine-tuning Adapter LoRA BitFit Prompt Tuning SSF FacT AdaLoRA LST
Builds on: Ladder Side Tuning Adapter LoRA BitFit Prompt Tuning
Closest papers by meaning
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.86)
- UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling (ICLR 2024 · similarity 0.85)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.85)
- Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization (ICML 2024 · similarity 0.85)
- V-PETL Bench: A Unified Visual Parameter-Efficient Transfer Learning Benchmark (NeurIPS 2024 · similarity 0.84)
- Faster Parameter-Efficient Tuning with Token Redundancy Reduction (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.