Research map / Learning theory and algorithms / Kernels and regression theory
Escaping Parameter Space: Tight Generalization Bounds via Representation Quality
Niclas A Göring, Shuofeng Zhang, Branton DeMoss, Ard Louis
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a tight generalization bound for deterministic neural networks from last-hidden-layer representation quality, evaluated with a parameter-free k-NN classifier.
Problem
Generalization bounds for deterministic neural networks are typically loose because they rely on parameter-space complexity measures, posterior KL divergences, weight norms, or compression arguments.
Approach
Bounds generalization directly from the quality of the last hidden layer representations, using a parameter-free k-nearest-neighbor classifier as the evaluator, thereby avoiding parameter-space complexity measures.
Key results
- Trained from scratch, the bound reaches an 8.0 percent (per the abstract, truncated) error-level certificate, claimed to be tight.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| certify via a k-NN decoder on a pretrained representation instead of random init (ablation) | ResNet-18 transfer certificate · CIFAR-10 | certified bound (%): 65.8 -> 20.1 | Improved |
| scale the pretrained model from DINOv2-L (304M) to DINOv2-g (1.1B) (ablation) | k-NN transfer certificate · ImageNet val | certified bound (%): 23.1 -> 23.5 | No clear effect |
| use pretrained ViT representation instead of training ViT from scratch (ablation) | ViT-small certificate · CIFAR-100 | certified bound (%): 99.3 -> 30.5 | Improved |
| switch ReLU to SiLU activation (ablation) | ResNet-18 cross-fit certificate · CIFAR-100 | certified bound (%): 51.1 -> 58.6 | Hurt |
| switch ReLU to SiLU activation (ablation) | FFNN cross-fit certificate · MNIST | certified bound (%): 4.9 -> 4.8 | No clear effect |
| switch ReLU to SiLU activation (ablation) | ViT cross-fit certificate · CIFAR-10 | certified bound (%): 31.9 -> 32.3 | No clear effect |
| use ResNet-50 instead of ResNet-18 (ablation) | cross-fit certificate (ReLU) · CIFAR-100 | certified bound (%): 51.1 -> 44.6 | Improved |
| train a small ViT from scratch on CIFAR-100 instead of a ResNet (ablation) | cross-fit certificate · CIFAR-100 | certified bound (%): 44.6 -> 99.3 | Hurt |
| add a regularizer pushing same-class representations apart (destroy local class geometry) (ablation) | six-layer FFNN on SVHN · SVHN | test accuracy | Hurt |
| increase network width in ResNet-18 (ablation) | local class geometry term Q1 · CIFAR-10 width sweep | Q1 excess over widest network | Improved |
| increase width through the interpolation threshold (ablation) | decoder-network agreement term Q2 · CIFAR-10 width sweep | Q2 excess | Mixed |
| certify through representation rather than through weight noise | k-NN decoder vs network · CIFAR-10, Gaussian weight noise sigma 0.01 | accuracy (%): 67 -> 84 | Improved |
Experimental setup
Metrics: generalization bound tightness test error
Closest papers by meaning
- Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach (ICLR 2019 · similarity 0.88)
- Fantastic Generalization Measures are Nowhere to be Found (ICLR 2024 · similarity 0.88)
- PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization (NeurIPS 2022 · similarity 0.88)
- Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks (ICML 2025 · similarity 0.87)
- Slicing Mutual Information Generalization Bounds for Neural Networks (ICML 2024 · similarity 0.86)
- Generalization Bounds for Rank-sparse Neural Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
