Research map / Architectures and efficiency / RNNs and normalization
Efficient ConvBN Blocks for Transfer Learning and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Tune mode for ConvBN blocks that computes fused weights on the fly, matching Eval-mode gradients and accuracy while cutting memory and time for transfer learning and adversarial example generation.
Problem
ConvBN blocks have a trade-off: Eval mode is stable for transfer learning but memory- and compute-heavy, while Deploy mode (fused weights) is efficient but unstable for training.
Approach
Keep BatchNorm statistics and affine parameters but compute the transformed convolution weight and bias on the fly each iteration, so the forward and backward passes are mathematically identical to Eval mode while storing only the input and fused weight instead of the convolution output.
Key results
- On TLlib classification (ResNet50), Tune mode reduces memory by about 36.8% and time per iteration by 7.9-9.2% vs Eval mode with equal or better accuracy (e.g., CUB-200 83.20 vs 82.62).
- On COCO detection with pre-trained backbones, Tune mode keeps mAP within about 0.003 of Eval mode while cutting memory by 19.9-43.0%.
- In adversarial example generation, Tune mode gives 5-8% speedup and 30-45% GPU memory savings across six architectures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Eval mode with Tune mode | ResNet50 classification fine-tuning · CUB-200 | memory (GB): 19.499 -> 12.323; 36.80% reduction | Helped |
| Replace Eval mode with Tune mode | ResNet50 classification fine-tuning · CUB-200 | time (second/iteration): 0.549 -> 0.501; 8.74% reduction | Helped |
| Replace Eval mode with Tune mode | ResNet50 classification fine-tuning · CUB-200 | accuracy: 82.62 (± 0.14) -> 83.20 (± 0.00) | No clear effect |
| Replace Eval mode with Tune mode | ResNet50 classification fine-tuning · Aircrafts | accuracy: 85.21 (± 0.22) -> 85.90 (± 0.26) | No clear effect |
| Replace Eval mode with Tune mode | ResNet50 classification fine-tuning · Stanford Cars | accuracy: 90.11 (± 0.03) -> 90.13 (± 0.12) | No clear effect |
| Replace Eval mode with Tune mode | object detectors on COCO · COCO | mAP: 0.3739 -> 0.3728 (-0.0011); 0.3824 -> 0.3825 (+0.0001); 0.3755 -> 0.3756 (+0.0001); 0.3675 -> 0.3647 (-0.0028); 0.3944 - | No clear effect |
| Replace Eval mode with Tune mode | object detectors on COCO · COCO | memory (GB): 22.15%↓; 19.85%↓; 27.08%↓; 23.59%↓; 27.65%↓; 31.62%↓; 28.06%↓; 35.76%↓; 43.04%↓ | Helped |
| Replace Eval mode with Tune mode | adversarial example generation · Adversarial example generation | time: 5%-8% speedup | Helped |
| Replace Eval mode with Tune mode | adversarial example generation · Adversarial example generation | memory (GB): 30%-45% | Helped |
| Train ConvBN blocks in Deploy mode instead of Eval mode | ResNet50 classification fine-tuning · CUB-200 | accuracy: 82.62 -> 62.96 | Hurt |
| Train ConvBN blocks in Deploy mode instead of Eval mode | ResNet50 classification fine-tuning · Aircrafts | accuracy: 85.21 -> 41.22 | Hurt |
| Train ConvBN blocks in Deploy mode instead of Eval mode | ResNet50 classification fine-tuning · Stanford Cars | accuracy: 90.11 -> 47.42 | Hurt |
Surprising findings
- Training in Deploy mode markedly hurts accuracy (e.g., CUB-200 62.96 vs 82.62 for Eval), showing fused-weight training is unstable.
- Eval mode can outperform Train mode for detection transfer learning (Faster RCNN ResNet101 mAP 0.3944 vs 0.3708).
Limitations
- Speedups in time are modest (roughly 5-10%), and object detection reports only memory savings.
- Experiments use V100 GPUs and mostly single-seed runs for detection due to cost.
- Pre-training experiments show slight accuracy drop when Tune mode is used in late stages.
Experimental setup
Datasets: CUB-200 Stanford Cars Aircrafts COCO ImageNet
Benchmarks: CUB-200 Stanford Cars Aircrafts COCO
Metrics: accuracy mAP GPU memory (GB) time per iteration (s)
Baselines: Eval mode Train mode Deploy mode FrozenBatchNorm Inplace-ABN
Builds on: Deploy mode BatchNorm fusion Inplace-ABN FrozenBatchNorm
Closest papers by meaning
- Diverse Branch Block: Building a Convolution as an Inception-Like Unit (CVPR 2021 · similarity 0.85)
- Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs (ICLR 2021 · similarity 0.84)
- LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (NeurIPS 2022 · similarity 0.84)
- LION: Implicit Vision Prompt Tuning (AAAI 2024 · similarity 0.83)
- Compute Better Spent: Replacing Dense Layers with Structured Matrices (ICML 2024 · similarity 0.83)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.