Research map / Adversarial robustness and security / Certified and empirical robustness
Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RobustDLRT, a dynamical low-rank training scheme with a condition-number-based spectral regularizer on the low-rank core S, improving adversarial robustness of compressed networks (~94% compression) without hurting clean accuracy.
Problem
Compressed (low-rank) neural networks are more sensitive to adversarial perturbations than full-rank baselines, and compression and adversarial robustness conflict.
Approach
Adds a regularizer R(S)=||S^T S - alpha^2 I|| on the r x r core of an orthonormal low-rank factorization inside rank-adaptive dynamical low-rank training, which controls the condition number of each layer and thereby its sensitivity to input perturbations; it extends soft orthogonality to well-conditioned low-rank cores and extends to convolutions via matricization.
Key results
- On UCM with VGG16 at 95.84% compression, RobustDLRT reaches 78.68% under l2-FGSM eps=0.1 versus 76.40% for baseline and 72.41% for DLRT.
- On CIFAR10 VGG16 under l1-FGSM eps=0.004, RobustDLRT (beta=0.15, 94.35% c.r.) gets 66.02% vs 64.66% baseline and 59.61% DLRT, outperforming listed literature methods.
- On ImageNet ViT-32l, RobustDLRT improves top-1 under l2-FGSM eps=0.3 to 25.24% from 15.47% for DLRT (about +9 points), with ~3% training runtime overhead over DLRT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spectral robustness regularization to DLRT | VGG16 · UCM, ℓ²-FGSM ε=0.3 | accuracy: 43.39$\pm$4.88 -> 53.30$\pm$3.14 | Helped |
| Add spectral robustness regularization to DLRT | VGG16 · UCM, Mixup ε=0.75 | accuracy: 20.13$\pm$2.92 -> 47.31$\pm$2.78 | Helped |
| Add spectral robustness regularization to DLRT | VGG16 · CIFAR10, ℓ²-FGSM ε=0.3 | accuracy: 28.74$\pm$2.21 -> 32.77$\pm$2.04 | Helped |
| Add spectral robustness regularization to DLRT | VGG16 · CIFAR10, Mixup ε=0.75 | accuracy: 38.98$\pm$1.94 -> 81.26$\pm$2.15 | Helped |
| Add spectral robustness regularization to DLRT | ViT-32l · ImageNet-1k, ℓ²-FGSM ε=0.3 | Top-1 accuracy: 15.47 -> 25.24 | Helped |
| Add spectral robustness regularization to DLRT | VGG16 · UCM clean | accuracy: 93.92$\pm$0.23 -> 94.61$\pm$0.35 | Helped |
| Add spectral robustness regularization to DLRT | ViT-32l · ImageNet-1k clean Top-1 | Top-1 accuracy: 72.27 -> 72.25 | No clear effect |
| Use spectral regularization during full-rank baseline training | VGG16 · UCM clean and adversarial validation | accuracy | Mixed |
| Use RobustDLRT with adversarial training instead of unregularized DLRT | VGG16 · UCM, adversarial training with ℓ²-FGSM | accuracy | Helped |
| Regularize the low-rank coefficient matrix while maintaining orthonormal U,V | LoRA-style training · VGG16, UCM | adversarial accuracy | Helped |
Surprising findings
- Regularizing the full baseline weight matrices improves adversarial robustness but reduces clean accuracy, unlike the low-rank core regularization.
- Regularized LoRA (non-orthonormal factors) yields lower clean and adversarial accuracy than DLRT-based regularized training, indicating orthonormality matters.
Limitations
- Improvements are not uniform: for some attacks (e.g., l2-FGSM eps=0.3 on VGG16) RobustDLRT remains below the baseline.
- Theoretical condition number bound depends on the smallest singular value being non-negligible; regularizer is non-convex so stability guarantees are only partial.
- Evaluated only on image classification with VGG and ViT models; robustness gains on other attacks like PGD only partially compared.
Experimental setup
Datasets: UCM CIFAR10 ImageNet-1k
Benchmarks: UCM CIFAR10 ImageNet-1k
Metrics: clean accuracy adversarial accuracy compression rate top-1 accuracy top-5 accuracy
Baselines: DLRT Baseline full-rank training LoRA Cayley SGD Projected SGD CondLR SVD prune
Builds on: dynamical low-rank training soft orthogonality regularizer LoRA Jitter attack Mixup attack
Closest papers by meaning
- Robust low-rank training via approximate orthonormal constraints (NeurIPS 2023 · similarity 0.90)
- On the Interaction of Compressibility and Adversarial Robustness (ICLR 2026 · similarity 0.88)
- Approximation to Smooth Functions by Low-Rank Swish Networks (ICML 2025 · similarity 0.86)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.86)
- COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation (NeurIPS 2025 · similarity 0.85)
- Low Rank Gradients and Where to Find Them (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.