Research map / Architectures and efficiency / Pruning and efficient tuning
Quantum-PEFT: Ultra parameter-efficient fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Quantum-PEFT, which parameterizes LoRA-style weight updates via quantum-circuit-inspired Pauli (RY/CZ) unitary parameterizations, giving logarithmic trainable-parameter scaling with full-rank updates.
Problem
LoRA-based PEFT has trainable parameters growing linearly with weight dimension and a rank-limited update, limiting parameter efficiency for large pretrained models.
Approach
Reparameterize low-rank factors U and V as unitary/Stiefel matrices generated by Pauli parameterization (alternating RY rotations and CZ entanglers, Kronecker-structured) or Taylor-series Lie-algebra mapping, so the parameter count grows logarithmically with dimension while the effective update is full-rank; quantum Shannon decomposition handles non-power-of-two dimensions.
Key results
- On GLUE with DeBERTaV3-base, Quantum-PEFT reaches 86.42 average using 0.013M trainable parameters, versus 86.12 for LoRA at 0.33M (about 25x fewer).
- On E2E with GPT-2 Medium, Quantum-PEFT (0.098M parameters) gets BLEU 67.46 vs LoRA 66.88 at 0.39M parameters.
- On ViT CIFAR-10 transfer with 3-bit base model, Quantum-PEFT reaches 98.46% accuracy with 0.007M parameters vs LoRA K=1 at 98.14% with 0.037M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Quantum-PEFT instead of AdaLoRA | DeBERTaV3-base · GLUE MRPC | accuracy: 89.46 -> 90.78 | Helped |
| Use Quantum-PEFT instead of AdaLoRA | DeBERTaV3-base · GLUE CoLA | Matthews correlation: 70.04 -> 67.85 | Hurt |
| Use Quantum-PEFT instead of LoRA | GPT-2 Medium · E2E | BLEU: 66.88 -> 67.46 | Helped |
| Use Quantum-PEFT instead of LoRA | GPT-2 Medium · E2E | ROUGE-L: 68.40 -> 67.36 | Hurt |
| Use Quantum-PEFT instead of AdaLoRA | Mistral-7B · GLUE SST-2 | accuracy: 96.90 -> 96.67 | Hurt |
| Use Quantum-PEFT instead of AdaLoRA | Mistral-7B · GLUE CoLA | Matthews correlation: 70.38 -> 70.61 | Helped |
| Use Quantum-PEFT instead of LoRA rank 4 | ViT · CIFAR10 | accuracy: 98.39% -> 98.46% | Helped |
| Use adaptive rather than uniform bit loading for INT1 Lie parameters (ablation) | Taylor parameterization · CIFAR10 | accuracy: 97.96% -> 98.64% | Helped |
| Reduce intrinsic rank from K'=8 to K'=1 (ablation) | Taylor parameterization, K=8 · CIFAR10 | accuracy: 98.87% -> 98.38% | Hurt |
| Increase entanglement layers from L=1 to L=3 (ablation) | Pauli parameterization · CIFAR10 with 2-bit base ViT | accuracy: 96.09% -> 96.71% | Helped |
| Increase entanglement layers from L=3 to L=32 (ablation) | Pauli parameterization · CIFAR10 with 2-bit base ViT | accuracy: 96.71% -> 96.71% | No clear effect |
Surprising findings
- Quantum-PEFT with 0.013M parameters outperforms AdaLoRA (0.32M) on average GLUE score for DeBERTaV3-base in the reported table, despite far fewer parameters.
- 1-bit quantization of Lie parameters with adaptive bit loading shows only 0.17% loss from FP32 in the ViT experiment.
Limitations
- Pauli parameterization has higher unitarity error than other mappings, per the appendix analysis.
- Results are not claimed to always beat LoRA; experiments limited to tasks with under 100k training instances due to compute limits.
- Ablation and sensitivity results show performance gains saturate with entanglement layers L, and optimal L is task-dependent.
Experimental setup
Datasets: GLUE E2E CIFAR-10 ImageNet-21k
Benchmarks: GLUE E2E CIFAR-10
Metrics: accuracy Matthews correlation Pearson/Spearman correlation BLEU NIST METEOR ROUGE-L CIDEr number of trainable parameters memory
Baselines: Full fine-tuning LoRA AdaLoRA BitFit Houlsby adapter Pfeiffer adapter LoHa LoKr MORA QuanTA
Builds on: LoRA AdaLoRA orthogonal fine-tuning (OFT) quantum two-design ansatz quantum Shannon decomposition
Closest papers by meaning
- QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation (NeurIPS 2024 · similarity 0.89)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.85)
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.85)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.84)
- F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning (NeurIPS 2025 · similarity 0.84)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.