Research map / Architectures and efficiency / Pruning and efficient tuning
Rethinking Token Reduction with Parameter-Efficient Fine-Tuning in ViT for Pixel-Level Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DAR-TR, a PEFT-compatible token reduction method for ViTs that uses a learnable mask after the FFN plus a decoupled convolutional token compensator to preserve attention diversity and high-frequency details on pixel-level tasks.
Problem
Token reduction in ViTs for PEFT reduces attention diversity and high-frequency detail, hurting pixel-level tasks such as segmentation that need long token sequences.
Approach
Decouples a convolutional token compensator (operating only on patch tokens) from the PEFT module, and applies a learnable Gumbel-sigmoid token mask after the FFN (rather than before) so that FFN bias perturbations are blocked and training converges.
Key results
- On DUTS, the method reaches F_w 0.901 with 90.83 GFLOPs versus 0.906 for AdaptFormer without token reduction (118.70 GFLOPs).
- Versus the DyT token-reduction baseline at the same 0.5 activation rate, the method improves DUTS F_w from 0.859 to 0.895 and Kvasir F_w from 0.897 to 0.938.
- In zero-shot camouflaged object segmentation trained on DUTS, the method reaches S-measure 0.812 on CAMO and 0.812 on COD10K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed token reduction method instead of DyT | DINOv2-B · COD10K | weighted F-measure (Fβw): .796 -> .810 | Helped |
| Use the proposed token reduction method instead of DyT | DINOv2-B · ISIC 2017 | weighted F-measure (Fβw): .843 -> .871 | Helped |
| Use the proposed token reduction method instead of DyT | DINOv2-B · CAMO | weighted F-measure (Fβw): .839 -> .842 | Helped |
| Use the proposed token reduction method instead of AdaptFormer | DINOv2-B · DUTS | weighted F-measure (Fβw): .906 -> .895 | Hurt |
| Use the proposed token reduction method instead of VPT | DINOv2-B · CUHK | weighted F-measure (Fβw): .800 -> .793 | Hurt |
| Use the proposed token reduction method instead of AdaptFormer | DINOv2-B · Kvasir | weighted F-measure (Fβw): .935 -> .938 | Helped |
| Add distillation to token reduction with token compensator (ablation) | DINOv2-B · DUTS | weighted F-measure (Fβw): .895 -> .901 | Helped |
| Add distillation to token reduction with token compensator (ablation) | DINOv2-B · Kvasir | weighted F-measure (Fβw): .938 -> .940 | No clear effect |
| Add token compensator to token reduction without distillation (ablation) | DINOv2-B · DUTS | weighted F-measure (Fβw): .886 -> .895 | Helped |
| Add token compensator to token reduction without distillation (ablation) | DINOv2-B · Kvasir | weighted F-measure (Fβw): .899 -> .938 | Helped |
| Use token compensator rank 48 instead of rank 24 (ablation) | DINOv2-B · DUTS | weighted F-measure (Fβw): .878 -> .895 | Helped |
| Use token compensator rank 48 instead of rank 96 (ablation) | DINOv2-B · DUTS | weighted F-measure (Fβw): .896 -> .895 | No clear effect |
Surprising findings
- A higher token keep rate in fixed token reduction is associated with a larger loss of high-frequency components in the final blocks, which the authors attribute to uniformity among longer tokens.
Limitations
- Fixed-rank comparison is limited to DINOv2-B and a few datasets, with additional results only in supplementary material.
- Convergence without the post-FFN mask and distillation is nontrivial, requiring design choices.
- Distillation does not help uniformly; on Kvasir the non-distilled variant is slightly better on S- and E-measure.
Experimental setup
Datasets: DUTS CUHK COD10K CAMO Kvasir ISIC 2017
Benchmarks: DUTS CUHK COD10K CAMO Kvasir ISIC 2017
Metrics: weighted F-measure S-measure E-measure FLOPs FPS trainable parameters
Baselines: VPT AdaptFormer LoRA EVP DyT (Dynamic Tuning) Linear probing Decoder only CaMF GenSAM SINet ZoomNeXt CRNet
Builds on: DynamicViT Dynamic Tuning DINOv2 AdaptFormer VPT LoRA
Closest papers by meaning
- HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models (AAAI 2025 · similarity 0.87)
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.87)
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.87)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.87)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.