Research map / Label-efficient and robust learning / Recognition and long-tail

Input Margins Can Predict Generalization Too

AAAI 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Data augmentation and generalization bias · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that classical input margins do not predict generalization, but input margins constrained to a low-dimensional PCA subspace (constrained margins, via a modified DeepFool/Taylor approximation) achieve strong rank correlation with test accuracy on PGDL.

Problem

Input-space classification margins have not been shown to predict generalization of deep networks, unlike hidden-representation margins; the PGDL benchmark is used to test whether a complexity measure based on input margins can rank models by generalization gap.

Approach

Restrict the input-space boundary search to the span of the top principal components of the training data (high-utility directions) and compute the margin with a Taylor/DeepFool iteration in that subspace, with clipping to feature bounds, yielding a "constrained margin" that is predictive of generalization.

Key results

• Constrained margins reach average Kendall's rank correlation 0.66 across development tasks versus 0.24 for standard input margins and 0.51 for first-layer hidden margins.
• On the PGDL test set, constrained margins achieve average CMI 27.03, the highest among compared measures (next best 23.62 for Schiff et al.), and outperform all other measures on 4 of 8 tasks.
• Constrained margin computation takes about 1 to 2 minutes per model on an Nvidia A30.

Measured effects reported in the paper

Surprising findings

• Standard input margins show negative rank correlation with generalization on several tasks (e.g., Task 6 at -0.14 and FMNIST at -0.12), despite being intuitively expected to predict generalization.
• Constraining to low-utility principal components causes the predictive power to degrade, with Task 1 reaching negative correlations.

Limitations

• Predictive performance is weaker on some tasks (e.g., Task 5 at 0.23, Task 7 at 0.50), and no single hyperparameter explains the failures.
• Models of different depth exhibit separate margin-accuracy relationships, so a single relationship does not hold across depths in some tasks.
• Number of principal components is chosen by the Kneedle elbow heuristic, which is not always optimal (e.g., Task 5 benefits from ~270 components).

Experimental setup

Datasets: PGDL dataset (tasks 1-9) CIFAR10 SVHN CINIC10 OxFlowers OxPets FMNIST

Benchmarks: PGDL (tasks 1, 2, 4, 5 development; 6-9 test)

Metrics: Kendall's rank correlation Conditional Mutual Information (CMI)

Baselines: Standard input margin Hidden margin (first layer) Hidden margin (all layers) Natekar and Sharma DBI*LWM Natekar and Sharma MM and AM Chuang et al. kV-Margin and kV-GN-Margin Schiff et al. PCA Gi&Mi

Builds on: DeepFool Jiang et al. hidden margin complexity measure Elsayed et al. large margin loss Natekar and Sharma margin with Mixup Chuang et al. k-variance margin normalization

Closest papers by meaning

• Predicting the Generalization Gap in Deep Networks with Margin Distributions (ICLR 2019 · similarity 0.92)
• Measuring Generalization with Optimal Transport (NeurIPS 2021 · similarity 0.90)
• Predicting Deep Neural Network Generalization with Perturbation Response Curves (NeurIPS 2021 · similarity 0.88)
• Improved Sample Complexities for Deep Neural Networks and Robust Classification via an All-Layer Margin (ICLR 2020 · similarity 0.88)
• Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergence (ICLR 2023 · similarity 0.88)
• Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.86)