Research map / Architectures and efficiency / Pruning and efficient tuning
Faster Parameter-Efficient Tuning with Token Redundancy Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FPET, a plug-and-play differentiable token merging module inserted at a mid-layer of ViT to speed up inference and training of parameter-efficient tuning methods on VTAB-1K.
Problem
Parameter-efficient tuning methods inherit the inference latency and computation of large pre-trained backbones and often add overhead from adapters, limiting practical use in compute-intensive settings.
Approach
Applies a single token merging step at the middle transformer layer, with a learnable quantized adapter refining keys for similarity, and a fully differentiable bipartite matching (checkerboard split, sigmoid-thresholded soft matching with straight-through estimator, gradients stopped before the backbone) instead of non-differentiable ToMe matching.
Key results
- FPET-Bi-AdaptFormer reaches 77.0% average VTAB-1K accuracy (matching the original) with 21.7% lower inference time, 23.7% lower FLOPs, and 18.4% lower GPU memory.
- FPET-AdaptFormer cuts training time by 19.57% (5.62 to 4.52 s/it) on VTAB-1K.
- FPET-LoRA achieves 75.6% average accuracy with 19.8% inference time reduction and 40.3% GPU memory reduction versus full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FPET token redundancy reduction to PET methods | Bi-AdaptFormer · VTAB-1K | inference time: 2.62 -> 2.17 | Helped |
| Add FPET token redundancy reduction to PET methods | Bi-AdaptFormer · VTAB-1K | accuracy: 77.0 -> 77.0 | No clear effect |
| Add FPET token redundancy reduction to PET methods | LoRA · VTAB-1K | training time: 6.24 -> 5.03 (-19.39%) | Helped |
| Use bipartite soft matching at the 6th layer rather than across layers (ablation) | Bi-AdaptFormer · VTAB-1K | accuracy: -0.27 -> -0.18 | Helped |
| Add checkerboard token assignment to BSM at the 6th layer (ablation) | Bi-AdaptFormer · VTAB-1K | accuracy: -0.18 -> -0.17 | Helped |
| Add key-refinement adapter without gradient stopping (ablation) | Bi-AdaptFormer · VTAB-1K | accuracy: -0.17 -> -1.70 | Hurt |
| Stop gradients from refined matching from propagating to the backbone (ablation) | Bi-AdaptFormer · VTAB-1K | accuracy: -1.70 -> -0.05 | Helped |
| Apply token reduction at layer 6 instead of no merging (ablation) | AdaptFormer · VTAB-1K | accuracy: 76.22% | No clear effect |
| Apply token reduction at layer 4 instead of layer 6 (ablation) | AdaptFormer · VTAB-1K | accuracy: -0.57% | Hurt |
| Apply token reduction at layer 2 instead of layer 6 (ablation) | AdaptFormer · VTAB-1K | accuracy: -1.93% | Hurt |
| Use LTMP token reduction instead of FPET matching (ablation) | Bi-AdaptFormer · VTAB-1K | accuracy: -0.38 | Hurt |
| Use ToMe token reduction instead of FPET matching (ablation) | Bi-AdaptFormer · VTAB-1K | GPU memory: -0.21 | Hurt |
Surprising findings
- Adding the key refinement adapter without gradient stopping degrades accuracy by 1.70 points, attributed to a push-and-pull effect among tokens.
Limitations
- Evaluation is limited to VTAB-1K image classification with ViT backbones.
- Accuracy drops at earlier merging layers (e.g. 1.93% at layer 2) reflect trade-offs between efficiency and performance.
Experimental setup
Datasets: VTAB-1K
Benchmarks: VTAB-1K
Metrics: top-1 accuracy inference time (ms) FLOPs (G) GPU memory (GB) training time (s/it)
Baselines: Full fine-tuning Linear probing VPT-Deep NOAH BitFit SSF FacT-TT RepAdapter LoRA AdaptFormer Bi-LoRA Bi-AdaptFormer
Builds on: ToMe (bipartite soft matching) LoRA AdaptFormer LTMP RepAdapter
Closest papers by meaning
- TTE: Two Tokens Are Enough to Improve Parameter-Efficient Tuning (AAAI 2025 · similarity 0.88)
- SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · similarity 0.87)
- RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning (NeurIPS 2026 · similarity 0.85)
- Rethinking Token Reduction with Parameter-Efficient Fine-Tuning in ViT for Pixel-Level Tasks (CVPR 2025 · similarity 0.85)
- Sheared Backpropagation for Fine-tuning Foundation Models (CVPR 2024 · similarity 0.85)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.