Research map / Architectures and efficiency / RNNs and normalization
RepAn: Enhanced Annealing through Re-parameterization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RepAn, which cycles re-parameterization, branch expansion, BN restoration and training inside simulated annealing so that a re-parameterized CNN can be trained incrementally with lossless compression.
Problem
Simulated annealing restarts for training ignore structural changes across cycles and rely on a fixed network structure, so knowledge is not incrementally inherited; re-parameterization (Rep) is used only for deployment, not for improving training accuracy.
Approach
Treat Rep as a lossless compression step inside annealing: after each cycle, merge branches into a single layer, re-expand with new randomly initialized branches weighted by an attach rate λ, restore BN structure, and continue training, so knowledge is inherited across cycles with no extra inference parameters.
Key results
- Improves RepVGG-A1 on CIFAR-100 from 64.90% to 71.28% top-1 (+6.38%).
- Improves ImageNet top-1 by up to 2.80% over baselines; RepVGG-B3 reaches 81.60%, reported as state of the art among Rep methods.
- ResNet-50 on ImageNet improves from 76.31% to 77.76%, and downstream COCO mAP and Cityscapes mIoU also improve.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply RepAn instead of baseline training | RepVGG-A1 · CIFAR-100 | top-1 accuracy: 64.90 -> 71.28 | Helped |
| apply RepAn instead of baseline training | ResNet-18 (DBB) · CIFAR-100 | top-1 accuracy: 73.44 -> 77.97 | Helped |
| apply RepAn instead of baseline training | ResNet-50 · ImageNet | top-1 accuracy: 76.31 -> 77.76 | Helped |
| apply RepAn instead of baseline training | RepVGG-B3 · ImageNet | top-1 accuracy: 80.52 -> 81.60 | Helped |
| set KD proportion α to 0 (remove knowledge distillation) (ablation) | RepAn with RepVGG-A1 · CIFAR-100 | top-1 accuracy: 71.28 -> 66.22 | Hurt |
| use KD α=0.5 and temperature=4 instead of α=0.9 and temperature=10 (ablation) | RepAn with RepVGG-A1 · CIFAR-10 | top-1 accuracy: 92.01 -> 91.20 | Hurt |
| increase KD temperature from 10 to 20 (ablation) | RepAn with RepVGG-A1 · CIFAR-10 | top-1 accuracy: 92.01 -> 91.57 | Hurt |
| increase KD proportion α from 0.9 to 1.0 (ablation) | RepAn with RepVGG-A1 · CIFAR-10 | top-1 accuracy: 92.01 -> 91.79 | Hurt |
| increase parallel branches from 1 to 5 (ablation) | RepAn with RepVGG-A1 · CIFAR-100 | top-1 accuracy: 71.3 -> 74.2 | Helped |
| increase parallel branches from 1 to 5 (ablation) | RepVGG-A1 with KD · CIFAR-100 | top-1 accuracy: 67.2 -> 70.5 | Helped |
| increase parallel branches from 1 to 5 (ablation) | RepVGG-A1 baseline · CIFAR-100 | top-1 accuracy: 64.9 -> 68.3 | Helped |
| increase parallel branches from 1 to 5 (ablation) | MobileOne-S0 · ImageNet | top-1 accuracy: 70.9 -> 71.1 | Mixed |
Surprising findings
- Without KD, the method still improves accuracy (e.g., RepVGG-A1 CIFAR-100 to 66.22% with alpha=0), and larger KD weights can hurt performance.
- Annealing with this method requires increased training overhead on ImageNet, which the authors attribute to capacity limits.
Limitations
- Annealing requires increased training overhead on larger datasets like ImageNet.
- Gains diminish with more parallel branches; no improvement on MobileOne ImageNet from more branches.
- Relies on KD or calibration choices; theoretical explanation is largely heuristic and toy-example based.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K MS-COCO Cityscapes
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K MS-COCO Cityscapes
Metrics: top-1 accuracy mAP mIoU
Baselines: KD ACNet DBB DyRep OREPA ODBB MobileOne
Builds on: RepVGG DBB ACNet DyRep snapshot ensemble
Closest papers by meaning
- RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks (ICLR 2025 · similarity 0.85)
- Make RepVGG Greater Again: A Quantization-Aware Approach (AAAI 2024 · similarity 0.84)
- RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation (NeurIPS 2025 · similarity 0.84)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.84)
- RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers (ICML 2025 · similarity 0.84)
- Gradient-Guided Annealing for Domain Generalization (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.