Research map / Architectures and efficiency / Pruning and efficient tuning
Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q3R, a quadratic reweighted rank regularizer derived from IRLS and a smoothed log-determinant rank surrogate, plus an AdamQ3R optimizer, to train neural network weights directly to low target ranks.
Problem
Low-rank pre-training of deep networks is hard: LoRA-style and post-hoc truncation methods lose accuracy when weights must stay low-rank while optimizing the task objective.
Approach
A majorizing quadratic model of an epsilon-smoothed log-determinant rank surrogate, whose reweighting operator is refreshed periodically via truncated SVD and whose smoothing parameter is lowered toward the target-rank singular value; optimized with a dedicated AdamQ3R that decouples the regularizer from the adaptive part, as AdamW does for weight decay.
Key results
- On CIFAR-10 ViT-Tiny, AdamQ3R retains 42.4% of parameters with a 1.22% accuracy drop and 23.2% with a 4.4% drop, while LoRITa underperforms.
- On GLUE with RoBERTa, Q3R-4 averages 85.86 versus 85.88 for dense fine-tuning and 84.58 for LoRA-4.
- On ViT-Base CIFAR-100 at 20% retained parameters, Q3R reaches 0.40-0.44 accuracy while LoRITa stays below 0.25.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Q3R instead of LoRITa D=3, λ=10^-1 | ViT-Tiny · CIFAR-10, 20% parameter retention | accuracy: 0.5035 -> 0.6550 | Helped |
| Use AdamQ3R instead of Adam with QKV regularization (ablation) | ViT-Base · ImageNet-1k validation, 20% parameter retention | accuracy: 0.4551 -> 0.4623 | Helped |
| Use AdamQ3R instead of Adam with QKV and MLP regularization (ablation) | ViT-Base · ImageNet-1k validation, 20% parameter retention | accuracy: 0.2950 -> 0.3376 | Helped |
| Regularize QKV only instead of QKV and MLP (ablation) | AdamQ3R on ViT-Base · ImageNet-1k validation, 20% parameter retention | accuracy: 0.3376 -> 0.4623 | Helped |
| Use Q3R-4 instead of LoRA-4 | RoBERTa fine-tuning · GLUE average | score: 84.58 -> 85.86 | Helped |
| Use Q3R-4 instead of dense fine-tuning | RoBERTa fine-tuning · GLUE average | score: 85.88 -> 85.86 | No clear effect |
| Increase target rank from 10% to 15% at λ=10^-3 (ablation) | Q3R on ViT-Tiny · CIFAR-10, 20% parameter retention | accuracy: 0.6295 -> 0.6215 | No clear effect |
| Increase λ from 10^-3 to 10^-2 at target rank 10% (ablation) | Q3R on ViT-Tiny · CIFAR-10, 20% parameter retention | accuracy: 0.6295 -> 0.6449 | Helped |
| Increase λ from 10^-3 to 10^-2 at target rank 15% (ablation) | Q3R on ViT-Tiny · CIFAR-10, 20% parameter retention | accuracy: 0.6215 -> 0.6550 | Helped |
| Increase λ from 10^-3 to 10^-2 at target rank 20% (ablation) | Q3R on ViT-Tiny · CIFAR-10, 20% parameter retention | accuracy: 0.6511 -> 0.6496 | No clear effect |
Surprising findings
- Q3R-regularized models outperform the unregularized baseline on ViT-Base ImageNet-1k in the paper's truncation experiments.
- Adding the Q3R terms into standard Adam is worse than a dedicated AdamQ3R optimizer.
Limitations
- Relies on choosing the regularization strength lambda and target rank; large lambda collapses to trivial very low-rank matrices.
- Weights remain dense during training, so no parameter reduction during training.
- Validated only at small-to-medium scale; viability on large models is not established.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k GLUE
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k GLUE
Metrics: accuracy post-truncation accuracy Matthews correlation F1
Baselines: LoRITa LoRA Vanilla ViT-T Adam Dense fine-tuning
Builds on: LoRITa LoRA AdamW IRLS
Closest papers by meaning
- QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks (NeurIPS 2025 · similarity 0.85)
- LRA-QViT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers (ICML 2025 · similarity 0.85)
- MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization (NeurIPS 2024 · similarity 0.84)
- From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications (ICML 2025 · similarity 0.84)
- Weight decay induces low-rank attention layers (NeurIPS 2024 · similarity 0.84)
- LoQT: Low-Rank Adapters for Quantized Pretraining (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.