Research map / Architectures and efficiency / CNN design
Pick-or-Mix: Dynamic Channel Sampling for ConvNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pick-or-Mix (PiX), a dynamic channel sampling module that replaces 1x1 channel-squeezing convolutions with per-pixel max/avg picks over channel subsets, serving as a channel squeezer, network downscaler, and dynamic channel pruner.
Problem
Channel pruning methods either deactivate channels (statically or dynamically) and need special implementations, while 1x1 channel-squeezing convolutions dominate computation and parameters in ConvNets.
Approach
PiX collapses the feature map to a per-channel global descriptor, predicts subset-wise sampling probabilities with a small linear layer, and then per pixel either takes the max (pick) or average (mix) of each channel subset, replacing dense 1x1 convolutions with a cheap, input-dependent, implementation-free operation.
Key results
- Replacing 1x1 squeezing layers in ResNet-50 with PiX cuts FLOPs by 22.8% (4.12B to 3.18B) while improving Top-1 from 76.30% to 76.77%.
- ResNet-18 + PiX reaches 70.60% top-1 at 1.85x FLOPs saving, versus 68.88% for the best listed dynamic pruning baseline (Manifold Reg. Pruning) at 2.06x.
- On ResNet-50, PiX reaches 79.40% top-1 at 160 epochs, above AFF (79.10%) and SKNet (79.21%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace 1x1 channel squeezing layers with PiX (zeta=4) | ResNet-50/101/152 · ImageNet | Top-1 accuracy and FLOPs: +0.27% to +0.47% Top-1 with ~23% fewer FLOPs | Helped |
| replace 1x1 channel squeezing layers with PiX (zeta=4) | ResNet family · GPU inference speed (Table 4) | FPS: 8% to 32% higher FPS | Helped |
| PiX with zeta=8 squeezing on ResNet-50 (ablation) | ResNet-50 · ImageNet | Top-1 accuracy: +0.81% over zeta=8 baseline with ~56% fewer FLOPs | Helped |
| add PiX channel squeezing to SE-equipped ResNet-50 (ablation) | ResNet-50 + SE · ImageNet | Top-1 accuracy and FLOPs: +0.10% Top-1 with ~23% fewer FLOPs | Helped |
| use PiX as downscaler (increase zeta) instead of width scaling | ResNet-18, ResNet-50, VGG-16, MobileNet-v1 · ImageNet | Top-1 accuracy at matched parameters: e.g. ResNet-18 at 6.77M: 70.60 vs 69.37; VGG-16 at 8.65M: 72.47 vs 70.53 | Helped |
| apply PiX to FFN of EfficientViT-M5 | EfficientViT-M5 · ImageNet | Top-1 accuracy and training hours: Top-1 +0.1 (76.8 to 76.9) with training time 36h to 24h | Mixed |
| PiX as dynamic channel pruner on ResNet-18 (vs prior dynamic pruning) | ResNet-18 · ImageNet | Top-1 accuracy: PiX 70.60 downscaled vs best prior dynamic pruner 70.71 (FBS) at roughly similar FLOPs saving; paper claims ~3% relative | Helped |
| PiX on ResNet-18 applied only before second layer of block (partial) vs full PiX (ablation) | ResNet-18 · ImageNet | Top-1 accuracy: full PiX 73.15 vs partial 71.65 | Helped |
| replace Max/Avg fusion operator with Min operator (ablation) | PiX · ImageNet | Top-1 accuracy: roughly 2% degradation | Hurt |
| remove sampling probability predictor (DWP, no 1x1 squeeze layers) (ablation) | ResNet-50 · ImageNet | Top-1 accuracy: accuracy loss of 1.30% relative to PiX channel squeezing | Hurt |
| PiX as channel squeezer on ResNet-50 (transfer) | ResNet-50 · CIFAR-100 transfer | Top-1 accuracy: 81.60 to 82.22 | Helped |
| PiX as backbone of segmentation model (ResNet-101 + PiX) | ResNet-101 segmentation · CityScapes | mIoU: 78.4 to 79.1 mIoU | Helped |
Surprising findings
- Replacing the 1x1 squeeze layers with PiX improves accuracy while reducing FLOPs, e.g. ResNet-101 + PiX surpasses the larger ResNet-152 baseline.
- Applying the FBS dynamic pruning design as a channel squeezer fails to converge, which the authors attribute to channel index loss.
Limitations
- Designed only for discrete squeezing factors zeta; non-integer sampling factors are not supported.
- Gains on Vision Transformers (EfficientViT FFN) are limited, with mainly runtime improvement and no accuracy gain at 0.5x scale.
- Transfer experiments and segmentation are limited to a few settings (CIFAR-10/100, CityScapes with one segmentation method).
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 CityScapes
Benchmarks: ImageNet CIFAR-10 CIFAR-100 CityScapes
Metrics: Top-1 accuracy FLOPs parameters training time inference FPS Top-5 error change
Baselines: SE CBAM FBS AFF SKNet DWP RepVGG-A0 Soft Filter Pruning Manifold Reg. Pruning Dynamic Struct. Pruning
Builds on: SENet FBS RepVGG EfficientViT
Closest papers by meaning
- BilevelPruning: Unified Dynamic and Static Channel Pruning for Convolutional Neural Networks (CVPR 2024 · similarity 0.86)
- CHEX: CHannel EXploration for CNN Model Compression (CVPR 2022 · similarity 0.85)
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (ICLR 2018 · similarity 0.85)
- Dynamic Channel Pruning: Feature Boosting and Suppression (ICLR 2019 · similarity 0.85)
- REPrune: Channel Pruning via Kernel Representative Selection (AAAI 2024 · similarity 0.84)
- KernelWarehouse: Rethinking the Design of Dynamic Convolution (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.