Research map / Architectures and efficiency / RNNs and normalization
Dr2Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dr2Net, a dual-residual reversible surrogate network that initializes from pretrained non-reversible models and dynamically anneals two residual coefficients to cut finetuning memory while keeping accuracy.
Problem
End-to-end finetuning of large pretrained vision models is memory-intensive for high-resolution or high-dimensional tasks (video understanding, small object detection, point cloud analysis), and existing reversible networks cannot directly reuse non-reversible pretrained weights.
Approach
Add a second reversible residual pathway to the pretrained architecture, weighted by coefficient beta, alongside the original residual weighted by alpha; start at alpha=1, beta=0.1 (equivalent to the pretrained model) and linearly transition to alpha≈0.3, beta≈0.7 so the network becomes numerically precise and reversible for memory-saving backprop.
Key results
- On temporal action detection (ActivityNet-v1.3), Dr2Net reaches 36.3% average mAP at 24.1GB versus 36.2% for conventional end-to-end finetuning at 44.7GB (46.1% memory saving).
- On action recognition with VideoMAE, Dr2Net achieves 89.0% top-5 accuracy at 6.0GB versus 90.3% for conventional finetuning at 29.3GB (79.5% saving).
- On point cloud segmentation, Dr2Net gives 68.1% mIoU at 15.6GB versus 69.6% for conventional finetuning at 22.5GB; on DINO object detection it gives 51.3% mAP at 30.0GB versus 54.0GB conventional.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Dynamically update α and β instead of keeping α = 1 and β = 0.1 fixed (ablation) | Dr2Net with Video Swin-tiny · ActivityNet-v1.3 | Avg. mAP: 35.44% -> 36.27% | Helped |
| Dynamically update α and β instead of keeping α = 1 and β = 0.1 fixed (ablation) | Dr2Net with VideoMAE video ViT-small · SthSth-v2 | Top-1 Acc: 26.48% -> 64.57% | Helped |
| Dynamically update α and β instead of keeping α = 1 and β = 0.1 fixed (ablation) | Dr2Net with MAE ViT-small · S3DIS | mIoU: 57.57% -> 68.13% | Helped |
| Update α and β simultaneously rather than α first and β second (ablation) | Dr2Net with VideoMAE video ViT-small · SthSth-v2 | Top-1 Acc: 60.51% -> 64.57% | Helped |
| Update α and β simultaneously rather than β first and α second (ablation) | Dr2Net with VideoMAE video ViT-small · SthSth-v2 | Top-1 Acc: 58.40% -> 64.57% | Helped |
| Update α and β simultaneously rather than α first and β second (ablation) | Dr2Net with Swin-tiny · MS-COCO | mAP: 50.2% -> 51.3% | Helped |
| Update α and β simultaneously rather than β first and α second (ablation) | Dr2Net with Swin-tiny · MS-COCO | mAP: 50.7% -> 51.3% | Helped |
| Update α and β simultaneously rather than α first and β second (ablation) | Dr2Net with MAE ViT-small · S3DIS | mIoU: 66.40% -> 68.13% | Helped |
| Update α and β simultaneously rather than β first and α second (ablation) | Dr2Net with MAE ViT-small · S3DIS | mIoU: 64.90% -> 68.13% | Helped |
| Use a smaller coefficient update interval η (ablation) | Dr2Net with VideoMAE video ViT-small · SthSth-v2 | Top-5 Acc: 89.01 (2 iter) -> 88.08 (100 iter) | Helped |
| Use linear rather than exponential coefficient schedules (ablation) | Dr2Net with Video Swin-tiny · ActivityNet-v1.3 | Avg. mAP: 35.46% -> 36.27% | Helped |
| Use linear rather than logarithm coefficient schedules (ablation) | Dr2Net with Video Swin-tiny · ActivityNet-v1.3 | Avg. mAP: 35.69% -> 36.27% | Helped |
Surprising findings
- Vanilla finetuning with fixed alpha=1, beta=0.1 on VideoMAE action recognition is worse than training the reversible model from scratch.
- Reversible training with Dr2Net was slower per epoch than conventional finetuning for some tasks (e.g., 261 vs 198 min for TAD, 89 vs 50 min for action recognition).
Limitations
- Reversible training adds roughly 33% more operations, increasing training time on some tasks.
- Requires beta>0 which makes the network only approximately equal to the pretrained model, causing numerical-error-driven accuracy gaps (e.g., vanilla finetuning is much worse).
- Experiments use only the smallest network variants due to limited compute, so memory-saving benefits at larger scale are not measured.
Experimental setup
Datasets: ActivityNet-v1.3 A2D-Sentences SthSth-v2 S3DIS MS-COCO Kinetics
Benchmarks: ActivityNet-v1.3 A2D-Sentences SthSth-v2 S3DIS MS-COCO
Metrics: mAP mIoU top-1 accuracy top-5 accuracy GPU memory (GB) training time per epoch
Baselines: Conventional end-to-end finetuning Frozen backbone Re2TAL reversible backbone (from scratch) Hard finetune VSGN MTTR Pix4Point DINO
Builds on: RevNet Re2TAL RevViT VSGN DINO
Closest papers by meaning
- Simplified Reversible Residual Networks (NeurIPS 2026 · similarity 0.90)
- The Reversible Residual Network: Backpropagation Without Storing Activations (NeurIPS 2017 · similarity 0.89)
- Multi-level Residual Networks from Dynamical Systems View (ICLR 2018 · similarity 0.88)
- Reversible Architectures for Arbitrarily Deep Residual Neural Networks (AAAI 2018 · similarity 0.87)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.85)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.