Research map / Architectures and efficiency / Pruning and efficient tuning

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Sparse Spectral Training (SST), a memory-efficient pre-training method that keeps all singular values trainable, samples singular vector pairs by multinomial weighting, and periodically re-SVDs, outperforming LoRA and ReLoRA* on Euclidean and hyperbolic networks.

Problem

Memory-efficient pre-training: LoRA is limited to a low-rank subspace and ReLoRA-style methods suffer saddle-point (zero-gradient) issues after each merge, leaving a large gap to full-rank training.

Approach

Parameterize each linear layer via SVD (U, Sigma, V^T), update all singular values every step, update only a multinomially sampled subset of singular vector pairs per iteration (swapping active/frozen segments), and re-SVD periodically so low-rank factors start from the current weight's spectrum rather than zero or random initialization.

Key results

• On LLaMA-1.3B, SST reaches perplexity 14.59 vs full-rank 14.54 (LoRA 16.50, ReLoRA* 17.32), reducing the gap by 97.4% with 18.7% of parameters trainable.
• On IWSLT'14 Euclidean Transformer (dim 64, r=8), SST achieves 22.28 BLEU vs LoRA 18.08 and ReLoRA* 18.12 (full 24.27); average BLEU gap reduction of 66.7%.
• On OPT-125M OpenWebText, SST reaches perplexity 26.98 vs full-rank 23.50, LoRA 34.23, ReLoRA* 35.80.

Measured effects reported in the paper

Surprising findings

• On the hyperbolic Transformer, SST outperforms full-rank training in several configurations (e.g., dimension 256).
• Restricting updates to top-r singular vectors (TOP_R) performs worst, while random, sequential, or multinomial sampling perform comparably when all vectors are eventually visited.

Limitations

• Optimizer states are reset at each iteration, requiring a warmup period.
• Parameter count is slightly higher than LoRA at the same rank due to the singular value vector.
• Evaluated mainly on small models (up to 1.3B parameters); LLM embeddings not covered.

Experimental setup

Datasets: OpenWebText C4 IWSLT'14 IWSLT'17 Multi30K Airport Cora Disease PubMed ImageNet

Benchmarks: OpenWebText validation perplexity C4 validation perplexity IWSLT'14 BLEU IWSLT'17 BLEU Multi30K BLEU 16 zero-shot NLP tasks from OPT paper node classification F1 (Airport, Cora, Disease, PubMed) link prediction precision

Metrics: perplexity BLEU accuracy F1 precision perplexity gap reduction

Baselines: Full-rank training LoRA ReLoRA* GaLore PiSSA

Builds on: LoRA ReLoRA GaLore PiSSA HyboNet

Closest papers by meaning

• Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.87)
• RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.86)
• ReLoRA: High-Rank Training Through Low-Rank Updates (ICLR 2024 · similarity 0.86)
• The Primacy of Magnitude in Low-Rank Adaptation (NeurIPS 2025 · similarity 0.86)
• Efficient Learning with Sine-Activated Low-Rank Matrices (ICLR 2025 · similarity 0.85)
• Expanding Sparse Tuning for Low Memory Usage (NeurIPS 2024 · similarity 0.85)