Research map / Architectures and efficiency / Pruning and efficient tuning
TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning
Qifeng Lei, Zhiyong Yang, Qianqian Xu, Cong Hua, Peisong Wen, Qingming Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LoRA with TuckA (multi-expert Tucker adapter, r=8,k=2) | DeBERTa-v3-base · GLUE average | average: 86.63 -> 88.73 | Improved |
| use TuckA r=2,k=4 (0.16M) vs LoRA (1.19M) | DeBERTa-v3-base · GLUE average | average: 86.63 -> 88.56 | Improved |
| replace HRA with TuckA r=8,k=2 | DeBERTa-v3-base · GLUE MRPC | accuracy/F1: 92.87 -> 92.82 | Hurt |
| replace HRA with TuckA r=6,k=1 | ViT-Base 10-shot · CIFAR100 | accuracy: 83.45 -> 84.81 | Improved |
| use TuckA instead of HRA/DoRA for GPU memory | ViT-Base CIFAR100 · peak GPU memory | MB: HRA 9530 -> TuckA 4514 (LoRA 4419) | Improved |
| replace LoRA with TuckA r=32,k=4 | Llama2-7B · MATH | accuracy: 6.62 -> 9.02 | Improved |
| replace LoRA r=32 with TuckA r=16,k=4 (fewer params) | Llama2-7B · GSM-8K | accuracy: 50.34 -> 50.34 | No clear effect |
| increase number of expert groups (ablation) | TuckA · CIFAR-100 few-shot | accuracy | Improved |
| increase rank to very large values (ablation) | TuckA · CIFAR-100 few-shot | accuracy | Mixed |
| unrestrict HRA to single-group TuckA (ablation) | TuckA vs HRA · CIFAR100 few-shot | accuracy | Improved |
| use data-aware initialization instead of Kaiming for router centroids (ablation) | TuckA · CIFAR-100 | expert load balance | Improved |
Closest papers by meaning
- PaCA: Partial Connection Adaptation for Efficient Fine-Tuning (ICLR 2025 · similarity 0.88)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.88)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.87)
- Decoupling Angles and Strength in Low-rank Adaptation (ICLR 2025 · similarity 0.87)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.87)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
