Research map / Architectures and efficiency / Pruning and efficient tuning
LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LORS, a low-rank residual parameter-sharing scheme for stacked decoders (shared plus per-layer low-rank private weights, static and adaptive), applied to AdaMixer on COCO to cut decoder parameters by ~70% with comparable AP.
Problem
Stacked identical layers (e.g., transformer decoders) cause large parameter counts, which hinders practical training and deployment.
Approach
Decompose each stacked layer's weight into a cross-layer shared matrix plus layer-specific low-rank residuals (B·A, optionally generated adaptively from the query), trained from scratch rather than for fine-tuning, so most parameters are shared while performance is maintained.
Key results
- With ResNet-50, 100 queries, 12 epochs, AdaMixer+LORS reduces decoder params from 110M to 35M (~70%) with AP 42.6 vs 42.7 for vanilla.
- Under 3x schedule with ResNet-50 and 100 queries, AdaMixer+LORS reaches 43.7 AP vs 43.2 for AdaMixer, with total params 60M vs 135M.
- With Swin-S and 300 queries (36 epochs), AdaMixer+LORS achieves 51.8 AP vs 51.3 with total params 85M vs 164M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LORS to the AdaMixer decoder | AdaMixer with ResNet-50, 100 queries, 12-epoch training · COCO 2017 val | AP: 42.7 -> 42.6 | No clear effect |
| Apply LORS to the AdaMixer decoder | AdaMixer with ResNet-50, 300 queries, 12-epoch training · COCO 2017 val | AP: 44.1 -> 44.1 | No clear effect |
| Apply LORS to the AdaMixer decoder | AdaMixer with ResNet-50, 100 queries, 36-epoch training · COCO 2017 val | AP: 43.2 -> 43.7 | Helped |
| Apply LORS to the AdaMixer decoder | AdaMixer with ResNet-50, 300 queries, 36-epoch training · COCO 2017 val | AP: 47.0 -> 47.6 | Helped |
| Apply LORS to the AdaMixer decoder | AdaMixer with R-101, 300 queries, 36-epoch training · COCO 2017 val | AP: 48.0 -> 48.2 | Helped |
| Apply LORS to the AdaMixer decoder | AdaMixer with Swin-S, 300 queries, 36-epoch training · COCO 2017 val | AP: 51.3 -> 51.8 | Helped |
| Add adaptive LORS only (ablation) | AdaMixer decoder, ResNet-50, 1× training · COCO 2017 val | AP: 42.5† -> 42.6 | No clear effect |
| Add static LORS only (ablation) | AdaMixer decoder, ResNet-50, 1× training · COCO 2017 val | AP: 42.5† -> 42.6 | No clear effect |
| Add shared parameters W shared (ablation) | LORS on AdaMixer ACM and ASM · COCO 2017 val | AP: 40.9 -> 42.6 | Helped |
| Add private parameters W private (ablation) | LORS on AdaMixer ACM and ASM · COCO 2017 val | AP: 41.4 -> 42.6 | Helped |
| Use increasing parameter groups 1 1 2 2 3 3 instead of 1 1 1 1 1 1 (ablation) | LORSA on AdaMixer decoder, rank 16 · COCO 2017 val | AP: 42.1 -> 42.6 | Helped |
| Set LORSA rank to 32 instead of 16 with one group per layer (ablation) | LORSA on AdaMixer decoder · COCO 2017 val | AP: 42.1 -> 41.2 | Hurt |
Surprising findings
- LORS improves AP over the vanilla model under longer 3x training despite using far fewer parameters, which the authors attribute to implicit regularization.
- Removing the private (layer-specific) weights hurts AP more (40.9) than removing shared weights (41.4), despite private weights being smaller.
Limitations
- Slightly increased inference time and GFLOPs due to serial low-rank computations.
- Longer training needed for LORS to reach its performance potential.
- Evaluated only on object detection with AdaMixer; not tested on NLP or language models.
Experimental setup
Datasets: MS COCO 2017 train ImageNet-1k (backbone pretraining)
Benchmarks: MS COCO 2017 val
Metrics: AP AP50 AP75 APs APm APl parameters (M) GFLOPs
Baselines: AdaMixer
Builds on: LoRA AdaMixer Deformable DETR
Closest papers by meaning
- Maestro: Uncovering Low-Rank Structures via Trainable Decomposition (ICML 2024 · similarity 0.84)
- LAuReL: Learned Augmented Residual Layer (ICML 2025 · similarity 0.84)
- LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models (ICML 2024 · similarity 0.84)
- Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks (NeurIPS 2025 · similarity 0.83)
- LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging (ICML 2024 · similarity 0.83)
- Deep Edge Filter: Return of the Human-Crafted Layer in Deep Learning (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.