Research map / Architectures and efficiency / CNN design

Beyond Skip Connection: Pooling and Unpooling Design for Elimination Singularities

AAAI 2025 · Architectures and efficiency › CNN design · approach: Filter design and trained-filter analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Pool Skip (max pooling + max unpooling + 3x3 conv + skip connection) inserted between convolutions and ReLU, motivated by a Weight Inertia hypothesis, and reports small error reductions on CIFAR, segmentation, and 3D medical imaging.

Problem

Deep CNN training suffers from elimination singularities (consistent deactivation of nodes, zero weights and dying ReLU) that stall feature propagation and learning.

Approach

Insert a Pool Skip module (max pooling, max unpooling, 3x3 convolution, and skip connection) between convolution and ReLU; the authors argue it provides affine and dimensional compensation that shifts negative pre-activation regions and unblocks zero-weight gradients.

Key results

• On CIFAR-100, MobileNet top-1 error improved from 33.75% to 28.31% (-5.44) with Pool Skip; ResNet34 improved only by -0.50 (22.69 to 22.19).
• On BTCV with nnUNet, Dice rose from 81.52% to 82.47% and mIoU from 72.14% to 73.08%.
• On AMOS, V-Net Dice improved from 77.15% to 80.02%, while nnUNet Dice was essentially unchanged (89.75 to 89.78).

Measured effects reported in the paper

Surprising findings

• Pool Skip gave no gain or a slight loss on ViT-based models (CVT CIFAR-10 error rose by 0.38), and on nnUNet AMOS NSD dropped by 0.06.
• In ablation on VGG16, removing skip connections worsened the error by about 13 points, while conv-only plus skip gave only a 3-point drop.

Limitations

• Adds many parameters since each convolution is followed by a 3x3 convolution, increasing training and inference cost.
• Max pooling requires feature map sizes divisible by the pool size, which fails for odd-sized encoder feature maps.
• Gains on ViT-based models are mixed and sometimes negative.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 Cityscapes Pascal VOC BTCV AMOS

Benchmarks: CIFAR-10 CIFAR-100 Cityscapes Pascal VOC BTCV AMOS

Metrics: top-1 error mIoU Dice NSD l2/l1 sparsity ratio

Baselines: MobileNet GoogLeNet VGG16 ResNet18 ResNet34 DeepLabv3+ V-Net nnUNet ViT CCT CVT

Builds on: ResNet Batch Normalization DeepLabv3+ nnUNet V-Net

Closest papers by meaning

• Skip Connections Eliminate Singularities (ICLR 2018 · similarity 0.87)
• Detail-Preserving Pooling in Deep Networks (CVPR 2018 · similarity 0.87)
• S3Pool: Pooling With Stochastic Spatial Sampling (CVPR 2017 · similarity 0.86)
• Upping the Game: How 2D U-Net Skip Connections Flip 3D Segmentation (NeurIPS 2024 · similarity 0.86)
• SkipPool: Improved Sparse Hierarchical Graph Pooling with Differentiable Exploration (AAAI 2025 · similarity 0.85)
• The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling (ICML 2024 · similarity 0.85)