Research map / Large language models / Instruction and fine-tuning
QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QuanTA, a quantum-circuit-inspired PEFT method that parameterizes high-rank weight updates via two-axis tensor operations on reshaped hidden states, merged at inference, outperforming LoRA and DoRA on commonsense and arithmetic reasoning with ~0.03-0.2% trainable parameters.
Problem
LoRA's low-rank weight updates may be insufficient for complex downstream tasks, limiting fine-tuning performance relative to full fine-tuning, while existing parameter-efficient methods remain constrained to low-rank representations.
Approach
Reshape the hidden dimension into N qudit-like axes and apply a sequence of trainable two-axis tensors, analogous to a quantum circuit of two-qubit gates, to parameterize full-rank (high-rank) weight updates; a frozen copy of the tensors at initialization guarantees zero initial update, and the result merges into the base weight with no inference overhead.
Key results
- On commonsense reasoning with LLaMA2-7B, QuanTA reaches 81.6 average accuracy vs. 77.6 for LoRA and 81.2 for full fine-tuning, using 0.041% trainable parameters.
- On arithmetic reasoning with LLaMA2-7B, QuanTA averages 80.5 (excluding AQuA) vs. 79.6 for LoRA and 79.3 for full fine-tuning, using 0.19% parameters.
- On LLaMA2-13B commonsense, QuanTA averages 84.8 vs. 82.3 for LoRA (0.029% vs 0.67% params).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QuanTA instead of the strongest listed baseline | LLaMA-7B · Commonsense reasoning (Avg.) | accuracy: 78.4 -> 80.3 | Helped |
| Use QuanTA instead of the strongest listed baseline | LLaMA-13B · Commonsense reasoning (Avg.) | accuracy: 81.5 -> 83.1 | Helped |
| Use QuanTA instead of full fine-tuning | LLaMA2-7B · Commonsense reasoning (Avg.) | accuracy: 81.2 -> 81.6 | Helped |
| Use QuanTA instead of LoRA | LLaMA2-13B · Commonsense reasoning (Avg.) | accuracy: 82.3 -> 84.8 | Helped |
| Use QuanTA instead of DoRA | LLaMA3-8B · Commonsense reasoning (Avg.) | accuracy: 85.2 -> 85.8 | Helped |
| Use QuanTA instead of LoRA | LLaMA2-7B · Arithmetic reasoning (Avg. W/O AQuA) | accuracy: 79.6 -> 80.5 | Helped |
| Use QuanTA instead of LoRA | LLaMA2-13B · Arithmetic reasoning (Avg. W/O AQuA) | accuracy: 82.5 -> 83.9 | Helped |
| Use QuanTA instead of LoRA | LLaMA2-7B · AQuA | accuracy: 17.5 -> 16.7 | Hurt |
| Use QuanTA instead of LoRA | LLaMA2-13B · AQuA | accuracy: 16.7 -> 18.9 | Helped |
Surprising findings
- QuanTA exceeds full fine-tuning on arithmetic reasoning tasks, which the authors attribute possibly to overfitting or hyperparameter difficulties for full fine-tuning.
- Subspace similarity analysis shows LoRA updates on DROP have high intrinsic rank while RTE has low intrinsic rank.
Limitations
- Sequential application of small tensors may underutilize GPUs, slowing implementation efficiency.
- Hyperparameters such as the number of tensors and axis decomposition were not optimized.
- Experiments only on LLaMA model series.
Experimental setup
Datasets: DROP Commonsense170K Math10K BoolQ PIQA SIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OBQA AQuA
Benchmarks: DROP BoolQ PIQA SIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OBQA AQuA GSM8K MAWPS
Metrics: F1-score accuracy
Baselines: LoRA DoRA full fine-tuning series adapters parallel adapters prefix tuning GPT-3.5
Builds on: LoRA DoRA KronA
Closest papers by meaning
- Quantum-PEFT: Ultra parameter-efficient fine-tuning (ICLR 2025 · similarity 0.89)
- RILQ: Rank-Insensitive LoRA-Based Quantization Error Compensation for Boosting 2-Bit Large Language Model Accuracy (AAAI 2025 · similarity 0.87)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.87)
- LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models (ICLR 2024 · similarity 0.87)
- LoQT: Low-Rank Adapters for Quantized Pretraining (NeurIPS 2024 · similarity 0.87)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.