Research map / Architectures and efficiency / Pruning and efficient tuning
Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLoRA, a Tucker-style low-rank tensor adaptation PEFT method that keeps an N-dimensional core tensor with the same order as the original weight, so convolution kernels are updated without reshaping to 2D.
Problem
Existing PEFT methods such as LoRA are designed for 2D linear weights; applying them to higher-dimensional parameter spaces (e.g., 4D convolution kernels) requires reshaping that breaks the spatial structure of the original tensor.
Approach
Model the weight update for an N-dimensional parameter as a low-rank core tensor of the same order multiplied by per-mode projection matrices (Tucker form), so the update keeps the original topological structure; for linear layers this becomes A G B^T with an unconstrained core matrix G.
Key results
- On ConvNeXt-V2-L, FLoRA with 12.77M parameters reaches 52.5 average versus 45.6 for LoRA at 12.94M, and with 40.49M parameters reaches 54.7 versus 46.9 for LoRA at 51.78M.
- On GLUE with DeBERTaV3-base at 1.33M parameters, FLoRA averages 89.80 versus 88.17 for LoRA and 88.49 for DoRA.
- On LLaVA-1.5-7B with 4.63% parameters, FLoRA averages 67.8 versus 67.6 for DoRA and 66.9 for LoRA; on VTAB-1K with 0.32M parameters, FLoRA averages 77.3, the highest reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FLoRA instead of LoRA or DoRA for convolutional fine-tuning | ConvNeXt-V2-L · DOTA | mAP: 18.3 -> 37.3 | Helped |
| Use FLoRA instead of DoRA | InternViT-6B · ADE20K, 0.66% parameters | Avg.: 63.4 -> 64.6 | Helped |
| Use FLoRA instead of AdaLoRA | DeBERTaV3-base · GLUE | Avg.: 89.23 -> 89.21 | No clear effect |
| Use FLoRA instead of DoRA | DeBERTaV3-base · GLUE | Avg.: 88.49 -> 89.80 | Helped |
| Use FLoRA instead of DoRA | LLaVA-1.5-7B · Visual instruction tuning | Avg.: 67.6 -> 67.8 | Helped |
| Use FLoRA instead of DoRA | LLaMA3-8B · Commonsense reasoning | Avg.: 85.0 -> 85.6 | Helped |
| Use FLoRA instead of LoRA | ViT-B/16 · VTAB-1K | Avg. accuracy (%): 76.4 -> 77.3 | Helped |
| Vary the FLoRA rank (ablation) | FLoRA · ConvNeXt-V2-L and DeBERTaV3-base downstream tasks | performance | Mixed |
| Vary the FLoRA scaling factor within a reasonable range (ablation) | FLoRA · ConvNeXt-V2-L and DeBERTaV3-base downstream tasks | performance | No clear effect |
Surprising findings
- On DOTA with ConvNeXt-V2-L, LoRA and DoRA perform much worse than the frozen base model (e.g., LoRA 18.3 mAP vs base 31.4), while FLoRA improves it to 37.3.
Limitations
- The scaling factor s must be tuned per backbone (e.g., different values for ConvNeXt-V2-L, InternViT-6B, DeBERTaV3-base), with no unified scale.
- Reported gains on many tasks are small (e.g., under 1 point on some GLUE and multimodal settings), with single-setting comparisons.
Experimental setup
Datasets: MS COCO ADE20K DOTA GLUE VQAv2 GQA VisWiz SQA TextVQA POPE MMBench Commonsense170K
Benchmarks: MS COCO ADE20K DOTA GLUE VQAv2 GQA VisWiz SQA TextVQA POPE MMBench Commonsense170K
Metrics: mAP mAcc mIoU accuracy Matthews correlation Pearson/Spearman correlation training time GPU memory Frobenius norm feature amplification factor
Baselines: Fully fine-tuning BitFit HAdapter PAdapter AdaLoRA LoRA DoRA
Builds on: LoRA Tucker decomposition DoRA AdaLoRA
Closest papers by meaning
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.87)
- Accurate and Efficient Low-Rank Model Merging in Core Space (NeurIPS 2025 · similarity 0.87)
- A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models (ICML 2025 · similarity 0.86)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.86)
- LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection (NeurIPS 2026 · similarity 0.85)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.