Research map / Architectures and efficiency / Pruning and efficient tuning
Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Coeff-Tuning, a PEFT method that views multi-head attention as a graph convolution over a filter subspace and tunes a tiny H×H matrix of combination coefficients over pre-trained attention maps, with residual parameterization and coefficient dropout, plug-and-play with LoRA/DoRA/SSF.
Problem
Existing parameter-efficient fine-tuning methods mostly adapt linear weight matrices and do not consider the expressiveness of the multi-head attention layer, whose softmax attention maps restrict outputs to the convex hull of value embeddings.
Approach
Reformulate multi-head attention as a graph convolution with a filter subspace of attention maps, then learn an unconstrained H×H coefficient matrix that linearly combines these maps (residual-parameterized, with dropout), expanding the output space beyond the convex hull while adding only H² parameters per layer.
Key results
- On VTAB-1k with ViT-B/16, Coeff. α Only reaches 69.78 mean accuracy with 0.002M params vs. LoRA 72.91 at 0.305M; Coeff-Tuning with SNELL-8 reaches 74.70 mean.
- On multi-modal VL-BART, Ours + DoRA reaches 77.7 average vs. DoRA 77.4 and full model tuning 77.3.
- Ablation: dropout p=0.2 gives best average score 77.7 vs 75.1 at p=0 on the multimodal task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Coeff-Tuning to SNELL-8 | SNELL-8 with ViT-B/16 · VTAB-1k | mean accuracy: 74.17 -> 74.70 | Helped |
| Add Coeff-Tuning to SNELL-8 | SNELL-8 with ViT-B/16 · FGVC | mean accuracy: 91.5 -> 91.9 | Helped |
| Add Coeff-Tuning to DoRA | DoRA with SDXL · SDXL concept customization | Text Align.: 0.33 -> 0.34 | Helped |
| Add Coeff-Tuning to DoRA | DoRA with SDXL · SDXL concept customization | Concept Align.: 0.73 -> 0.75 | Helped |
| Add Coeff-Tuning to DoRA | DoRA with SDXL · SDXL concept customization | Diversity: 10.9 -> 11.37 | Helped |
| Add Coeff-Tuning to LoRA | LoRA with SDXL · SDXL concept customization | Text Align.: 0.33 -> 0.34 | Helped |
| Add Coeff-Tuning to LoRA | LoRA with SDXL · SDXL concept customization | Concept Align.: 0.74 -> 0.75 | Helped |
| Add Coeff-Tuning to LoRA | LoRA with SDXL · SDXL concept customization | Diversity: 10.08 -> 10.45 | Helped |
| Add Coeff-Tuning to DoRA | DoRA with VL-BART · Image-text understanding | Avg.: 77.4 -> 77.7 | Helped |
| Use residual parameterization I + α instead of direct α (ablation) | Coeff-Tuning with ViT-B/16 · Few-shot image classification | Avg. Score: 44.28 -> 73.49 | Helped |
| Use residual parameterization I + α instead of direct α (ablation) | Coeff-Tuning with ViT-B/16 · Few-shot image classification | Avg. Score: 61.57 -> 73.49 | Helped |
| Set coefficient dropout rate to 0.2 instead of 0.0 (ablation) | Coeff-Tuning with VL-BART · Multi-modal fine-tuning | Avg. Score: 75.1 -> 77.7 | Helped |
Surprising findings
- Tuning only the tiny H×H coefficient matrix (about 0.002M params) already outperforms several PEFT baselines such as LoRA and Adapter on VTAB-1k.
Limitations
- Gains over strongest baselines are often small (e.g., 77.7 vs 77.4 on VL-BART; 73.64 vs 73.04 on VTAB with FACT)
- Expressiveness analysis is supported mainly by a toy single-layer validation experiment
- Dropout rate must be tuned per setting (p=0 to 0.4 gave varying results)
Experimental setup
Datasets: VTAB-1k CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars DreamBooth CustomConcept101 VQAv2 GQA NLVR2 MSCOCO captions
Benchmarks: VTAB-1k FGVC CustomConcept101/DreamBooth concept customization VQAv2 GQA NLVR2 COCO Captions
Metrics: accuracy CLIP concept alignment CLIP text alignment Vendi diversity score VQAv2/GQA/NLVR2 accuracy CIDEr
Baselines: Full fine-tuning Linear Probing Adapter BitFit VPT LoRA SPT-Adapter FACT-TT SNELL-8 SSF DoRA Full Model
Builds on: LoRA DoRA SSF VPT VL-BART
Closest papers by meaning
- Large Convolutional Model Tuning via Filter Subspace (ICLR 2025 · similarity 0.90)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.85)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.85)
- Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Data (NeurIPS 2026 · similarity 0.85)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.85)
- LoKi: Low-dimensional KAN for Efficient Fine-tuning Image Models (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.