Research map / Label-efficient and robust learning / Recognition and long-tail
Predicting the Generalization Gap in Deep Networks with Margin Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use normalized margin-distribution quartiles across multiple layers with log-transformed features | 9-layer CNN generalization-gap predictor · CIFAR-10 | adjusted R^2: 0.72 -> 0.94 | Helped |
| Use normalized margin-distribution quartiles across multiple layers with log-transformed features | ResNet-32 generalization-gap predictor · CIFAR-10 | adjusted R^2: 0.44 -> 0.87 | Helped |
| Use normalized margin-distribution quartiles across multiple layers with log-transformed features | ResNet-32 generalization-gap predictor · CIFAR-100 | adjusted R^2: 0.95 -> 0.97 | Helped |
| Add margin normalization (ablation) | Multi-layer margin predictor · CIFAR-10, CNN | adjusted R^2: 0.89 -> 0.94 | Helped |
| Add margin normalization (ablation) | Multi-layer margin predictor · CIFAR-10, ResNet | adjusted R^2: 0.82 -> 0.87 | Helped |
| Add margin normalization (ablation) | Multi-layer margin predictor · CIFAR-100, ResNet | adjusted R^2: 0.95 -> 0.97 | Helped |
| Use log-transformed rather than linear features (ablation) | Normalized multi-layer margin predictor · CIFAR-10, CNN | adjusted R^2: 0.88 -> 0.94 | Helped |
| Use log-transformed rather than linear features (ablation) | Normalized multi-layer margin predictor · CIFAR-10, ResNet | adjusted R^2: 0.77 -> 0.87 | Helped |
| Use log-transformed rather than linear features (ablation) | Normalized multi-layer margin predictor · CIFAR-100, ResNet | adjusted R^2: 0.91 -> 0.97 | Helped |
| Use margin-distribution features from all sampled layers instead of one layer (ablation) | Normalized multi-layer margin predictor · CIFAR-10, ResNet | adjusted R^2: 0.44 -> 0.87 | Helped |
| Use quartile statistics rather than only one statistic per layer (ablation) | Normalized multi-layer margin predictor · CIFAR-10, ResNet | adjusted R^2: 0.53 -> 0.87 | Helped |
| Use quartile statistics rather than the first five moments (ablation) | Normalized multi-layer margin predictor · CIFAR-10, CNN | adjusted R^2: 0.93 -> 0.94 | No clear effect |
Closest papers by meaning
- Input Margins Can Predict Generalization Too (AAAI 2024 · similarity 0.92)
- Measuring Generalization with Optimal Transport (NeurIPS 2021 · similarity 0.92)
- Large Margin Deep Networks for Classification (NeurIPS 2018 · similarity 0.89)
- Fantastic Generalization Measures and Where to Find Them (ICLR 2020 · similarity 0.89)
- Improved Sample Complexities for Deep Neural Networks and Robust Classification via an All-Layer Margin (ICLR 2020 · similarity 0.89)
- Fantastic Generalization Measures are Nowhere to be Found (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.