Research map / Architectures and efficiency / RNNs and normalization

Geometry-aware training of factorized layers in tensor Tucker format

NeurIPS 2024 · Architectures and efficiency › RNNs and normalization · approach: Efficient network compression and training · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TDLRT, a rank-adaptive, geometry-aware dynamical low-rank training method that trains convolutional/tensor layers directly in Tucker format, with loss-descent, convergence and approximation guarantees.

Problem

Training neural network layers in low-rank factorized form requires a priori rank choice, warm-up with the full model, and is sensitive to initialization; direct factor training is ill-conditioned due to manifold curvature.

Approach

Applies dynamical low-rank approximation on the manifold of Tucker tensors using a change of variables K_i = U_i S_i so that the gradient flow has no stiff pseudoinverse term; each step uses one gradient evaluation with basis augmentation, then core update and HOSVD-based rank truncation controlled by a single tolerance tau.

Key results

• On CIFAR10 with VGG16, TDLRT reaches 90.23% test accuracy at 94.40% compression rate versus 92.01% for the full baseline.
• On CIFAR10 with AlexNet, TDLRT reaches 82.39% accuracy at 83.12% compression, outperforming the other factorization baselines (best other: TT-factorized 78.13%).
• On GLUE with DeBERTa V3 fine-tuning, TDLRT with 0.9M parameters outperforms LoRA (1.33M params) on all five reported tasks, e.g., SST2 95.94 vs 94.84.

Measured effects reported in the paper

Surprising findings

• On ResNet18 CIFAR10, the TDLRT compression rate (78.73%) is below several baselines, while RGD achieves the best accuracy (92.76%) and TDLRT is only second-tier, indicating no uniform dominance.
• Tucker-factorized and CP-factorized baselines fall well below the full baseline at comparable compression, showing direct factor training is unstable.

Limitations

• Assumes well-performing low-rank Tucker subnetworks exist in the reference network, lacking theoretical support.
• Efficient GPU implementation requires an efficient tensor rounding algorithm; the rank truncation uses HOSVD.
• Compression is controlled by a single tolerance tau that must be tuned; strategies for choosing tau are left open.

Experimental setup

Datasets: CIFAR10 Tiny-ImageNet MNIST GLUE DreamBooth

Benchmarks: CIFAR10 Tiny-ImageNet MNIST GLUE

Metrics: test accuracy compression rate training loss wall-clock time to 60% accuracy

Baselines: Matrix DLRT Tucker-factorized Matrix-factorized CP-factorized Tucker RGD TT-factorized SNIP IMP GraSP LoRA

Builds on: dynamical low-rank approximation LoRA low-rank matrix DLRT

Closest papers by meaning

• Implicit Regularization in Tensor Factorization (ICML 2021 · similarity 0.88)
• Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.87)
• Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent (ICML 2025 · similarity 0.87)
• Factor Augmented Tensor-on-Tensor Neural Networks (AAAI 2025 · similarity 0.87)
• COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation (NeurIPS 2025 · similarity 0.86)
• Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices (NeurIPS 2024 · similarity 0.86)