Research map / Architectures and efficiency / RNNs and normalization
Layer Compression of Deep Networks with Straight Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compresses a deep residual/transformer network into a one-step straight-flow model via a continuous-time flow, ReFlow straightening, and final knowledge distillation, reducing depth and FLOPs with little accuracy loss.
Problem
Very deep networks are slow to infer and hard to deploy; the goal is to compress a trained deep network into a shallow (few- or one-step) network without losing accuracy.
Approach
Distill a deep network into a time-continuous neural flow, apply ReFlow to straighten transport trajectories so the flow can be discretized in one step, then refine the one-step model with knowledge distillation together with the head and tokenizer; the same flow model doubles as an early-exit dynamic network.
Key results
- On ImageNet, the compressed SWIN-Base reaches 82.6% top-1 at 5.1 GFLOPs vs 83.1% for the 15.1 GFLOPs original, outperforming the best baseline (82.0%).
- On ResNet-50 the one-step model keeps 77.0% top-1 at 1.86 GFLOPs (original 77.0% at 4.12 GFLOPs), beating KD (76.6%) and DBP (76.8%).
- On CIFAR-10 with DeiT-Base, the 4-layer compressed model reaches 95.4% vs 94.5% standard training at 34.2% FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use straight-flow compression instead of direct standard knowledge distillation | 4-layer student from DeiT-Base · CIFAR-10 test | Top-1 accuracy (%): 95.2±0.01 -> 95.4±0.01 | Helped |
| Use straight-flow compression instead of Growing | compressed SWIN-Base · ImageNet validation | Top-1 accuracy (%): 82.2±0.1 -> 82.6±0.2 | Helped |
| Use straight-flow compression instead of Growing | compressed DeiT-Base · ImageNet validation | Top-1 accuracy (%): 81.4±0.1 -> 81.6±0.1 | Helped |
| Use straight-flow compression instead of Growing | compressed ResNet-50 · ImageNet validation | Top-1 accuracy (%): 76.8±0.3 -> 77.0±0.1 | No clear effect |
| Use a one-step model instead of the standard model | SWIN-Base segmentation backbone · ADE20K test | mIoU (%): 48.1 -> 47.6 | Hurt |
| Use two inference steps instead of one | SWIN-Base segmentation backbone · ADE20K test | mIoU (%): 47.6 -> 48.1 | Helped |
| Use a one-step model instead of the standard model | DeiT-Base segmentation backbone · ADE20K test | mIoU (%): 44.2 -> 44.0 | Hurt |
| Use two inference steps instead of one | DeiT-Base segmentation backbone · ADE20K test | mIoU (%): 44.0 -> 44.2 | Helped |
| Replace the linear trajectory objective with cosine or DDPM trajectories (ablation) | DeiT-Base flow model · CIFAR-10 test, 1-Step | Top-1 accuracy (%): 95.4±0.01 vs 94.7±0.01; 94.9±0.01 | Hurt |
| Add one ReFlow operation to the initial flow (ablation) | DeiT-Base flow model · CIFAR-10 test, 1-Step | Top-1 accuracy (%): 94.9±0.02 -> 95.2±0.02 | Helped |
| Apply ReFlow twice instead of once (ablation) | DeiT-Base flow model · CIFAR-10 test, 1-Step | Top-1 accuracy (%): 95.2±0.02 -> 95.3±0.03 | Helped |
| Apply ReFlow twice instead of once (ablation) | DeiT-Base flow model · CIFAR-10 test, 2-Step | Top-1 accuracy (%): 95.5±0.02 -> 95.4±0.02 | Hurt |
Surprising findings
- On CIFAR-10, the 4-layer distilled model slightly exceeds the 12-layer DeiT-Base teacher's ImageNet-style reference accuracy of 95.5% at 2 steps only within noise; ReFlow twice gives slightly better one-step but slightly worse two-step accuracy, showing a trade-off.
Limitations
- Multi-resolution architectures need a separate flow model per resolution stage, so the method cannot weight resolutions adaptively.
- The flow architecture is manually designed; architecture search is left open.
- ReFlow cannot guarantee the training loss and accuracy are preserved; accuracy trades off against number of reflow iterations.
Experimental setup
Datasets: CIFAR-10 ImageNet ADE20K
Benchmarks: CIFAR-10 ImageNet ADE20K
Metrics: top-1 accuracy FLOPs mIoU FPS
Baselines: Standard KD Attention Transfer Feature Map Transfer Growing DBP LWP Cosine trajectory DDPM trajectory RL agent early exit Confidence early exit
Builds on: Rectified flow (ReFlow) Neural ODE TinyBERT MobileBERT growing distillation DBP
Closest papers by meaning
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (ICLR 2018 · similarity 0.88)
- Model compression via distillation and quantization (ICLR 2018 · similarity 0.88)
- Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts (NeurIPS 2020 · similarity 0.88)
- Model Preserving Compression for Neural Networks (NeurIPS 2022 · similarity 0.88)
- Auto-Compressing Networks (NeurIPS 2025 · similarity 0.86)
- LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.