Research map / Learning theory and algorithms / Kernels and regression theory

Rethinking Information-theoretic Generalization: Loss Entropy Induced PAC Bounds

ICLR 2024 · Learning theory and algorithms › Kernels and regression theory · approach: Information-theoretic generalization bounds

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes loss-entropy-induced information-theoretic PAC generalization bounds that avoid the bounded-loss assumption and apply to long-tailed loss distributions.

Problem

Information-theoretic generalization analysis is mostly limited to average-case guarantees and requires a bounded loss assumption, leaving computationally tractable PAC generalization analysis (especially for long-tailed losses) largely unaddressed.

Approach

Derives PAC-style generalization bounds induced by the entropy of the loss distribution (loss entropy) rather than relying on bounded-loss or average-case information measures, targeting long-tailed loss behavior.

Experimental setup

Builds on: information-theoretic generalization bounds

Closest papers by meaning

• Exactly Tight Information-theoretic Generalization Bounds via Binary Jensen-Shannon Divergence (ICML 2025 · similarity 0.92)
• Towards Generalization beyond Pointwise Learning: A Unified Information-theoretic Perspective (ICML 2024 · similarity 0.90)
• Tighter CMI-Based Generalization Bounds via Stochastic Projection and Quantization (NeurIPS 2025 · similarity 0.89)
• Fantastic Generalization Measures are Nowhere to be Found (ICLR 2024 · similarity 0.89)
• Generalizability of Neural Networks Minimizing Empirical Risk Based on Expressive Power (ICLR 2025 · similarity 0.88)
• Slicing Mutual Information Generalization Bounds for Neural Networks (ICML 2024 · similarity 0.88)