Research map / Learning theory and algorithms / Kernels and regression theory
Generalization and Robustness of the Tilted Empirical Risk
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives uniform and information-theoretic generalization bounds for the tilted empirical risk under negative tilt with unbounded losses (bounded (1+ε)-th moment), analyzes its robustness under distribution shift, proposes a data-driven tilt choice, and studies KL-regularized TERM via tilted Gibbs posteriors.
Problem
Generalization error of tilted empirical risk (TER) for unbounded losses and its robustness to noisy outliers and distribution shift; existing bounds for empirical risk need bounded losses or give worse rates.
Approach
Uses negative tilt with a (1+ε)-th moment assumption and Bernstein/variance bounds on exp(γℓ) to get O(n^{-ε/(1+ε)}) tilted generalization bounds; bounds robustness via a TV-distance term under distribution shift, and proposes a closed-form-like data-driven tilt from the bound.
Key results
- Pareto-outlier experiment (n=1000): data-driven tilt achieves lower population risk than ERM across outlier ratios (e.g., 0.01 vs 0.03 at 0.1% outliers).
- Theory gives tilted generalization error bounds with convergence rate O(n^{-ε/(1+ε)}) for unbounded losses under bounded (1+ε)-th moment and negative tilt.
- KL-regularized tilted Gibbs posterior achieves expected tilted generalization error bound with rate O(n^{-ε}) under conditions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use negative-tilt TERM instead of ERM | Logistic regression with Gaussian outliers · Gaussian outliers, 0.1%–70.0% | population risk: ERM -> data-driven TERM: 0.05 -> 0.00; 0.22 -> 0.16; 0.30 -> 0.20; 0.28 -> 0.14; 0.18 -> 0.15 | Helped |
| Use negative-tilt TERM instead of ERM | Logistic regression with Pareto outliers · Pareto outliers, 0.1%–70.00% | population risk: ERM -> data-driven TERM: 0.03 -> 0.01; 0.02 -> 0.01; 0.01 -> 0.01; 0.01 -> 0.01; 0.02 -> 0.01 | Helped |
| Use grid-search negative-tilt TERM instead of ERM | Logistic regression with Gaussian outliers · Gaussian outliers, 0.1%–70.0% | population risk: ERM -> grid-search TERM: 0.05 -> 0.00; 0.22 -> 0.15; 0.30 -> 0.16; 0.28 -> 0.11; 0.18 -> 0.14 | Helped |
| Use grid-search negative-tilt TERM instead of ERM | Logistic regression with Pareto outliers · Pareto outliers, 0.1%–70.00% | population risk: ERM -> grid-search TERM: 0.03 -> 0.00; 0.02 -> 0.00; 0.01 -> 0.00; 0.01 -> 0.01; 0.02 -> 0.00 | Helped |
Limitations
- Unbounded-loss bounds require a lower bound on sample size n.
- Information-theoretic bounds are non-vacuous only for bounded mutual information; positive tilt not analyzed.
- Experiments are small-scale synthetic logistic regression with three runs.
Experimental setup
Datasets: synthetic logistic regression data with Gaussian and Pareto outliers
Benchmarks: synthetic logistic regression with Gaussian outliers synthetic logistic regression with Pareto outliers
Metrics: population risk
Baselines: empirical risk minimization
Builds on: tilted empirical risk minimization Bernstein inequality information-theoretic generalization bounds Gibbs posterior
Closest papers by meaning
- Tilted Empirical Risk Minimization (ICLR 2021 · similarity 0.90)
- Tilted Sparse Additive Models (ICML 2023 · similarity 0.87)
- Understanding Domain Generalization: A Noise Robustness Perspective (ICLR 2024 · similarity 0.86)
- On the Asymptotic Distribution of the Minimum Empirical Risk (ICML 2024 · similarity 0.85)
- Achievable distributional robustness when the robust risk is only partially identified (NeurIPS 2024 · similarity 0.85)
- The Price of Implicit Bias in Adversarially Robust Generalization (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.