Research map / Label-efficient and robust learning / Recognition and long-tail

Catch-Up Mix: Catch-Up Class for Struggling Filters in CNN

AAAI 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Loss design and classifier theory · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Catch-up Mix, a feature-level mixup that swaps low-influence (low-norm, relative to a paired sample) filter activations to give under-trained "struggling" filters more learning opportunities, improving accuracy and robustness of CNN classifiers.

Problem

CNNs over-rely on a small subset of strong (high-norm, fast-learning) filters, while slow-learning filters receive vanishing gradients late in training, hurting generalization and robustness.

Approach

Feature-level mixup that, for a random layer, compares per-filter L2 norms (sum-normalized, relative filter influence) of paired samples and mixes in the filters with lower relative influence from the paired sample, so under-developed filters are trained while strong filters are masked out; CutMix is used when the input layer is selected.

Key results

• On Tiny-ImageNet with PreActResNet-18, Catch-up Mix reaches 30.73% top-1 error vs 32.98% for the best baseline (PuzzleMix), per the deformation table.
• On CIFAR-100 PreActResNet-18, Catch-up Mix achieves 17.76% top-1 error and 68.16% FGSM error, the best among compared mixup methods.
• On ImageNet-O with ResNet-50, Catch-up Mix achieves FPR95 81.92% vs 88.54% for RecursiveMix, while ImageNet top-1 accuracy is 78.71%.

Measured effects reported in the paper

Surprising findings

• Randomly mixing channels without the norm-based selection (RandomChannelMix) hurts Tiny-ImageNet accuracy (61.21 vs 63.02 baseline), showing the selection strategy matters.
• On ImageNet-1k, the method shows an accuracy-robustness trade-off, with top-1 accuracy (78.71%) below some mixup baselines such as RecursiveMix (79.14%).

Limitations

• Accuracy-robustness trade-off on large, data-rich settings such as ImageNet, where feature diversity can reduce accuracy.
• Gains are reported mostly relative to mixup baselines with reproduced numbers; the method's hyperparameter alpha needs dataset-specific tuning (0.5 for CIFAR-100, 10 for others).

Experimental setup

Datasets: CIFAR-100 Tiny-ImageNet CUB-200-2011 Stanford Cars FGVC-Aircraft ImageNet-1k CIFAR-100-C ImageNet-O

Benchmarks: CIFAR-100 Tiny-ImageNet CUB-200-2011 Stanford Cars FGVC-Aircraft ImageNet-1k ImageNet-O CIFAR-100-C

Metrics: top-1 accuracy FGSM error rate mean corruption error (mCE) deformation accuracy FPR95 AUROC AUPR GT-known localization training time per epoch

Baselines: InputMixup CutMix PuzzleMix SaliencyMix Co-Mixup SnapMix ManifoldMixup MoEx AlignMixup RecursiveMix

Builds on: mixup CutMix ManifoldMixup AlignMixup PuzzleMix

Closest papers by meaning

• On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks (NeurIPS 2019 · similarity 0.88)
• Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup (ICML 2023 · similarity 0.88)
• PatchUp: A Feature-Space Block-Level Regularization Technique for Convolutional Neural Networks (AAAI 2022 · similarity 0.88)
• Harnessing Hard Mixed Samples with Decoupled Regularizer (NeurIPS 2023 · similarity 0.88)
• Adversarial AutoMixup (ICLR 2024 · similarity 0.87)
• Pushing Boundaries: Mixup's Influence on Neural Collapse (ICLR 2024 · similarity 0.86)