Research map / Architectures and efficiency / Pruning and efficient tuning
Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CDRA-SPT, which reorganizes sensitive parameters into a compact sub-matrix by row/column and applies dynamic-rank LoRA-style adaptation to it, improving sensitivity-aware PEFT on vision tasks with fewer parameters.
Problem
Sensitivity-aware PEFT (SPT) evaluates sensitivity at the whole-matrix level and ignores the row/column concentration of sensitive parameters, causing storage overhead and limited gains in vision model fine-tuning.
Approach
Reorder rows and columns by sensitivity so sensitive weights form a compact sub-matrix, then apply low-rank adaptation to that sub-matrix with a per-matrix rank chosen from its size and sensitive-parameter count.
Key results
- On VTAB-1k with ViT-B/16, CDRA-SPT reaches 74.53% mean accuracy using 0.49% trainable parameters, versus 73.87% for SPT-LoRA-16 (1.36%).
- On FGVC, CDRA-SPT achieves 89.86% mean accuracy at 0.86% params, above SPT-LoRA-16 at 89.69% (2.28%).
- On ADE20K, CDRA reaches 74.0 mDice / 62.0 mIoU at 0.82% params, versus SPT-LoRA-16 at 73.3 / 61.4 (1.60%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CDRA-SPT instead of SPT-LoRA-16 | VTAB-1k fine-tuning · VTAB-1k | Mean Acc.: 73.87 -> 74.53 | Helped |
| Use CDRA-SPT instead of SPT-LoRA-16 | FGVC fine-tuning · FGVC | MeanAcc.: 89.69 -> 89.86 | Helped |
| Use CDRA-SPT instead of LoRA | DINOv2 semantic segmentation · ADE20K | mDice: 73.8 -> 74.0 | Helped |
| Use CDRA-SPT instead of SPT-LoRA-16 | DINOv2 semantic segmentation · VOC | mIoU: 66.3 -> 66.4 | Helped |
| Replace dynamic rank with fixed rank 8, 16, or 32 (ablation) | CDRA-SPT · VTAB-1k | Mean Acc: 76.8 -> 75.0; 75.1; 75.4 | Hurt |
| Apply CDRA to Case 1 only (ablation) | SPT · VTAB-1k | Mean Acc: 75.4 -> 75.7 | Helped |
| Apply CDRA to Case 2 only (ablation) | SPT · VTAB-1k | Mean Acc: 75.4 -> 75.8 | Helped |
| Apply CDRA to both Case 1 and Case 2 (ablation) | SPT · VTAB-1k | Mean Acc: 75.4 -> 76.0 | Helped |
| Expand compact sub-parameter matrix size to 1.5 times (ablation) | CDRA-SPT · VTAB-1k | Mean Acc: 76.8 -> 75.0 | Hurt |
| Reduce compact sub-parameter matrix size to 0.5 times (ablation) | CDRA-SPT · VTAB-1k | Mean Acc: 76.8 -> 74.6 | Hurt |
| Tune sub-parameter matrix △w00_i instead of △w10_i (ablation) | CDRA-SPT · VTAB-1k | Mean Acc: 76.8 -> 75.2 | Hurt |
| Tune sub-parameter matrix △w01_i instead of △w10_i (ablation) | CDRA-SPT · VTAB-1k | Mean Acc: 76.8 -> 74.8 | Hurt |
Surprising findings
- Applying the adaptation only to the sensitive-row/sensitive-column sub-matrix (Δw10) outperforms applying it to the other three sub-blocks, and the other sub-blocks are argued to be non-sensitive.
Limitations
- Dynamic-rank allocation rule (Eq. 7) is heuristic and the paper gives no theoretical justification for it.
- Sub-matrix size sensitivity: scaling the compact matrix to 0.5x or 1.5x reduces accuracy by about 1-2 points, indicating sensitivity to this hyperparameter.
- Experiments are limited to vision backbones (ViT-B/16 and DINOv2) and do not test other architectures or modalities.
Experimental setup
Datasets: VTAB-1k FGVC (CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, Stanford Cars) ADE20K PASCAL VOC
Benchmarks: VTAB-1k FGVC ADE20K VOC
Metrics: mean accuracy mDice mIoU trainable parameters (%) inference time memory
Baselines: Full fine-tuning VPT-Deep Adapter NOAH LoRA Bias GPS SPT-LoRA-8 SPT-LoRA-16
Builds on: SPT LoRA VPT GPS
Closest papers by meaning
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.90)
- Expanding Sparse Tuning for Low Memory Usage (NeurIPS 2024 · similarity 0.89)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.88)
- Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model (CVPR 2024 · similarity 0.88)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.88)
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.