Research map / Architectures and efficiency / Pruning and efficient tuning
Efficient Learning with Sine-Activated Low-Rank Matrices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sine-activated low-rank decomposition (sin(ω·UV^T)) that raises matrix rank without extra parameters, and shows it as a drop-in improvement for LoRA/DoRA on LLMs, ViT, ConvNeXt, NeRF and 3D occupancy.
Problem
Low-rank decompositions reduce parameter counts but their rank constraint limits representational capacity, causing accuracy loss relative to full-rank layers.
Approach
Apply an element-wise sine with a fixed high frequency ω to the low-rank product UV^T; a theorem shows this raises the rank above k for sufficiently large ω with no added trainable parameters, making it a plug-in for existing low-rank methods.
Key results
- On LLaMA 3-8B commonsense reasoning, Sine LoRA k=4 averages 83.61 vs LoRA k=16 82.9 and LoRA k=32 80.82 with fewer parameters.
- On NeRF LLFF at k=1 (1.3% of parameters), Sine Low-Rank reaches 19.77 average PSNR vs 14.00 for low-rank.
- On 3D occupancy Thai Statue with k=1 (2.1% params), IoU rises from 84.3 (low-rank) to 90.8 (sine low-rank).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply sine activation to LoRA k=1 | RoBERTa V3 · GLUE | Avg.: 0.32 | Helped |
| Apply sine activation to LoRA k=2 | RoBERTa V3 · GLUE | Avg.: 0.45 | Helped |
| Apply sine activation to LoRA k=4 | RoBERTa V3 · GLUE | Avg.: 0.73 | Helped |
| Apply sine activation to LoRA k=8 | RoBERTa V3 · GLUE | Avg.: 0.42 | Helped |
| Apply sine activation to LoRA k=4 | LLaMA 3-8B · Commonsense reasoning | Avg.: 3.57 | Helped |
| Apply sine activation to LoRA k=8 | LLaMA 3-8B · Commonsense reasoning | Avg.: 2.87 | Helped |
| Apply sine activation to LoRA k=16 | LLaMA 3-8B · Commonsense reasoning | Avg.: 2.45 | Helped |
| Apply sine activation to LoRA k=32 | LLaMA 3-8B · Commonsense reasoning | Avg.: 2.74 | Helped |
| Apply sine activation to DoRA k=8 | LLaMA 3-8B · Commonsense reasoning | Avg.: 1.4 | Helped |
| Apply sine activation to DoRA k=16 | LLaMA 3-8B · Commonsense reasoning | Avg.: 0.3 | Helped |
| Apply sine activation to DoRA k=32 | LLaMA 3-8B · Commonsense reasoning | Avg.: 0.1 | Helped |
| Apply sine activation to the low-rank matrix at k=1 | NeRF · LLFF | Average PSNR: 5.77 | Helped |
Surprising findings
- Sine low-rank ViT-Base at rank 250 with 50% of parameters reportedly matches the full-rank baseline on ImageNet-1k, and sine variants exceed full-rank on some settings.
- Sine LoRA at rank 4 outperforms standard LoRA at all higher ranks on LLaMA 3-8B.
Limitations
- Accuracy of very low-rank sine matrices still falls short of full-rank models, per the authors.
- No principled way to set frequency ω; it is tuned as a hyperparameter per rank and task.
- Sine activation increases training memory and compute overhead relative to plain low-rank methods.
Experimental setup
Datasets: GLUE CommonsenseQA-style commonsense reasoning suite (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OBQA) CIFAR-100 ImageNet-1k LLFF Stanford Thai Statue, Dragon, Lucy scans
Benchmarks: GLUE BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA CIFAR-100 ImageNet-1k LLFF
Metrics: accuracy average accuracy PSNR IoU parameter count compression rate BD-Rate BD-PSNR
Baselines: LoRA DoRA Low-Rank ViT Full-Rank NeRF ConvNeXt-Tiny
Builds on: LoRA DoRA Vision Transformer NeRF
Closest papers by meaning
- Maestro: Uncovering Low-Rank Structures via Trainable Decomposition (ICML 2024 · similarity 0.86)
- OTLRM: Orthogonal Learning-based Low-Rank Metric for Multi-Dimensional Inverse Problems (AAAI 2025 · similarity 0.86)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.86)
- Low Rank Gradients and Where to Find Them (NeurIPS 2025 · similarity 0.86)
- Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks (ICML 2025 · similarity 0.85)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.