Research map / Continual learning / Continual fine-tuning of LLMs
Low-Rank Adaptation in Multilinear Operator Networks for Security-Preserving Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoLoRA, a polynomial (multiplication-and-addition-only) low-rank adaptation for MONet combined with gradient projection memory, enabling exemplar-free class-incremental learning on encrypted-compatible networks.
Problem
Multilinear Operator Networks (MONet), which can run on data under Leveled Fully Homomorphic Encryption, suffer catastrophic forgetting in class-incremental learning, and standard PEFT methods (e.g., adapters with ReLU/GELU) violate LFHE constraints.
Approach
A low-rank update built only from multiplications and additions, applied to MONet channel projections (PoLoRA-C) or as parallel projections to Mu-Layers (PoLoRA-M); the down-projection matrices are initialized from gradient projection memory so new-task updates are orthogonal to old-task gradients, and only up-projections are trained per task.
Key results
- IPoLoRA-C reaches 62.85% average accuracy on ImageNet-R with 10 tasks, about 22 points above Classifier FT (42.97%).
- On CIFAR-100 with 10 tasks, IPoLoRA-C achieves 70.70% vs 49.74% for Classifier FT.
- IPoLoRA-M improves on PoLoRA-M by about 30% on average across settings, with nearly 39% gain on ImageNet-R with 20 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GPM initialization to PoLoRA-C (ablation) | MONet-S with PoLoRA-C · ImageNet-R (10 tasks) | average accuracy (%): 50.33 ± 0.91 -> 62.85 ± 0.61 | Helped |
| add GPM initialization to PoLoRA-M (ablation) | MONet-S with PoLoRA-M · ImageNet-R (10 tasks) | average accuracy (%): 28.72 ± 0.81 -> 60.09 ± 0.28 | Helped |
| use PoLoRA-M without GPM instead of classifier fine-tuning (ablation) | MONet-S · ImageNet-R (10 tasks) | average accuracy (%): 49.74 ± 0.30 -> 28.72 ± 0.81 | Hurt |
| set GPM scaling factor too high or too low (ablation) | IPoLoRA-M and IPoLoRA-C · ImageNet-R (10 tasks) and CIFAR-100 (10 tasks) | accuracy (%): 6.98; 60.10; 62.38; 50.41 | Mixed |
| reduce At rank (ablation) | IPoLoRA-M and IPoLoRA-C · CIFAR-100 and PACS | accuracy (%): 67.24; 69.05; 69.21; 68.02; 70.13; 70.79; 62.43; 65.87; 69.70; 70.40; 76.13; 77.56 | Mixed |
| apply PoLoRA to only selected channel projections instead of all (B, C, D) (ablation) | IPoLoRA-C · PACS | accuracy (%): 77.56 -> 76.98 | Hurt |
| apply PoLoRA to only selected channel projections instead of all (A, C) (ablation) | IPoLoRA-C · PACS | accuracy (%): 77.56 -> 73.44 | Hurt |
| apply PoLoRA to only selected channel projections instead of all (A, B, D) (ablation) | IPoLoRA-C · PACS | accuracy (%): 77.56 -> 75.84 | Hurt |
| apply PoLoRA only to channel projection C (ablation) | IPoLoRA-C · PACS | accuracy (%): 77.56 -> 68.91 | Hurt |
| use IPoLoRA-C rather than IPoLoRA-M | MONet-S · ImageNet-R (10 tasks) | average accuracy (%): 60.09 ± 0.28 -> 62.85 ± 0.61 | Helped |
Surprising findings
- Plain PoLoRA-M performs worse than Classifier FT in final average accuracy, showing that low-rank integration alone is insufficient without GPM initialization.
- Scaling the GPM initialization factor too high hurts IPoLoRA-C, and IPoLoRA-M requires very large scaling values to perform well.
Limitations
- Forgetting is only partially mitigated; accuracy drops across many tasks remain substantial.
- Performance of IPoLoRA-M is sensitive to the GPM scaling factor, requiring extremely large values.
- Evaluated only on image classification with a single MONet backbone size.
Experimental setup
Datasets: CIFAR-100 ImageNet-R PACS
Benchmarks: CIFAR-100 ImageNet-R PACS
Metrics: average accuracy after all tasks per-task accuracy accuracy vs number of known tasks
Baselines: Full Fine-tuning Classifier Fine-tuning PoLoRA-M PoLoRA-C
Builds on: MONet LoRA Gradient Projection Memory Adapter
Closest papers by meaning
- Multilinear Operator Networks (ICLR 2024 · similarity 0.84)
- Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption (ICML 2024 · similarity 0.82)
- Flora: Low-Rank Adapters Are Secretly Gradient Compressors (ICML 2024 · similarity 0.82)
- PoLAR: Polar-Decomposed Low-Rank Adapter Representation (NeurIPS 2025 · similarity 0.81)
- Privacy-Preserving Low-Rank Adaptation Against Membership Inference Attacks for Latent Diffusion Models (AAAI 2025 · similarity 0.81)
- Backdoor Channels Hidden in Latent Space: Cryptographic Undetectability in Modern Neural Networks (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.