Research map / Architectures and efficiency / Pruning and efficient tuning
PaCA: Partial Connection Adaptation for Efficient Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PaCA, a PEFT method that fine-tunes randomly selected partial columns of pretrained weights instead of adding adapters, cutting training time and activation memory versus LoRA while keeping comparable accuracy.
Problem
PEFT methods like LoRA reduce trainable parameters but not training time, because adapter layers are processed sequentially with pretrained layers on GPUs, and LoRA does not reduce activation memory during fine-tuning.
Approach
Fine-tune a random subset of columns of the pretrained weight matrices directly (no adapter layers), so forward/backward for trainable connections is fused into the pretrained layer's kernels and only the partial input activations need to be stored for weight gradients.
Key results
- On LLaMA3-8B with MMLU, PaCA (rank 16) reaches 65.4% average accuracy vs. 65.0% for LoRA while using 23G vs. 27G memory and 3.5h vs. 4.4h training time.
- PaCA increases maximum sequence length on a single A100 from 8.0K (LoRA) to 9.8K, and improves throughput by 16% over LoRA on both A100 and Gaudi2 HPU.
- QPaCA on LLaMA3.1-70B uses 69G vs. 80G memory for QLoRA and 4.7h vs. 5.1h training time, with MT-Bench average 6.08 vs. 6.09.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PaCA rank 8 instead of LoRA rank 8 | LLaMA2-7B fine-tuning · MMLU 5-shot | Avg. accuracy | No clear effect |
| Use PaCA rank 16 instead of LoRA-family baselines | LLaMA2-7B fine-tuning · MMLU 5-shot | Training time: 26% | Helped |
| Use PaCA rank 16 instead of DoRA | LLaMA2-7B fine-tuning · MMLU 5-shot | Memory: 31% | Helped |
| Increase PaCA rank from 8 to 16 (ablation) | LLaMA2-7B fine-tuning · MMLU 5-shot | Avg. accuracy: 50.4 -> 51.2 | Helped |
| Use PaCA rank 16 instead of LoRA-family baselines | LLaMA3-8B fine-tuning · MMLU 5-shot | Avg. accuracy: 65.4 vs 65.2 | Helped |
| Use PaCA rank 64 instead of LoRA | LLaMA3-8B instruction tuning on Oasst1 · MT-Bench | Training time: 19% | Helped |
| Increase PaCA rank from 64 to 128 (ablation) | LLaMA3-8B instruction tuning on Oasst1 · MT-Bench | Avg. score: 5.23 -> 5.26 | Helped |
| Increase PaCA rank from 64 to 128 (ablation) | LLaMA3-8B instruction tuning on Oasst1 · MT-Bench | Memory: 47G -> 51G | Hurt |
| Use QPaCA instead of QLoRA | LLaMA3.1-70B instruction tuning on Oasst1 · MT-Bench | Avg. score: 6.09 -> 6.08 | No clear effect |
| Use PaCA instead of LoRA | LLaMA3-8B fine-tuning · Maximum sequence length | Sequence length: 23% | Helped |
| Use PaCA instead of LoRA and its variants | LLaMA3-8B training on NVIDIA A100 and Intel Gaudi2 · Sequence length 512 throughput | Throughput: 16% | Helped |
| Use weight-based or gradient-based connection selection instead of random selection (ablation) | PaCA on LLaMA3-8B · MT-Bench (Oasst1 fine-tuning) | Avg. score: 5.23; 5.18; 5.24 | No clear effect |
Surprising findings
- Random selection of connections performs comparably to weight-norm or gradient-based selection strategies.
- Backward propagation in PaCA takes about 17% longer than forward despite near-identical FLOPs, attributed to sequential weight-gradient and input-gradient computation.
Limitations
- Relies on random column selection; no method for identifying optimal partial connections is provided.
- Convergence proof assumes Lipschitz continuous gradients, which may not hold for large networks.
- Accuracy is slightly lower than DoRA/MosLoRA at matched rank 8 on some tasks (e.g., up to 0.9% drop on LLaMA2-7B).
Experimental setup
Datasets: MMLU Oasst1 CIFAR-10 CIFAR-100 Oxford-IIIT Pets Oxford-Flowers 102
Benchmarks: MMLU MT-Bench CIFAR-10 CIFAR-100 Oxford-IIIT Pets Oxford-Flowers 102
Metrics: accuracy training time memory usage MT-Bench score max sequence length throughput (sentences/s)
Baselines: LoRA DoRA MosLoRA QLoRA Full-FT VeRA QA-LoRA
Builds on: LoRA QLoRA DoRA MosLoRA
Closest papers by meaning
- Sparse is Enough in Fine-tuning Pre-trained Large Language Models (ICML 2024 · similarity 0.86)
- F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning (NeurIPS 2025 · similarity 0.86)
- Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation (ICML 2025 · similarity 0.85)
- Spectral Adapter: Fine-Tuning in Spectral Space (NeurIPS 2024 · similarity 0.85)
- Sheared Backpropagation for Fine-tuning Foundation Models (CVPR 2024 · similarity 0.85)
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.