Research map / Probabilistic and scientific modeling / Variational and generative sampling
Cauchy-Schwarz Divergence Information Bottleneck for Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Deep information bottleneck methods for regression rely on KL-based terms that reduce to MSE under Gaussian assumptions and use variational or upper-bound approximations of mutual information, introducing bias and sub-optimality.
Approach
Formulate IB entirely with Cauchy-Schwarz divergence: a CS-divergence prediction term between p(y|x) and the model's predictive distribution, plus CS-QMI as an exact, non-parametric compression term estimated with kernel density Gram matrices, requiring no distributional assumptions or variational bounds.
Key results
- placeholder
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MSE prediction term with CS divergence prediction term (CS-IB at r=0) | CS-IB · six regression datasets | RMSE | Helped |
| Use CS-QMI compression term with CS divergence prediction term (full CS-IB) at r=0.5 | CS-IB · Housing, Energy, PM2.5, Bike, Rotation MNIST, UTKFace | RMSE at r=0.5: e.g. Housing 0.245 vs NIB 0.267; Rotation MNIST 3.930 vs 4.102 | Helped |
| Adversarial robustness: CS-IB vs other IB approaches under white-box FGSM and PGD attacks | CS-IB · six regression datasets | RMSE under attack: e.g. Energy PGD5 0.402 vs NIB 0.471 | Helped |
| Replacing MSE with CS divergence prediction term in NIB, VIB-style baselines (ablation) | NIB · ablation appendix | performance | Helped |
| Use HSIC-bottleneck and other deep IB baselines instead of CS-IB | HSIC-bottleneck · Housing RMSE at r=0.5 | RMSE: 0.371 vs 0.258 at r=0 | Hurt |
Surprising findings
- Even at compression ratio r=0 (no compression term), the CS divergence prediction term alone gives a clear gain over MSE-based training.
Limitations
- placeholder
Experimental setup
Datasets: California Housing Appliance Energy Beijing PM2.5 Bike Sharing Rotation MNIST UTKFace
Benchmarks: California Housing Appliance Energy Beijing PM2.5 Bike Sharing Rotation MNIST UTKFace
Metrics: RMSE compression ratio information plane I(y;t)
Baselines: VIB NIB Square-NIB Exp-NIB HSIC-bottleneck
Builds on: variational information bottleneck nonlinear information bottleneck HSIC bottleneck squared-IB convex-IB
Closest papers by meaning
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.91)
- Structured IB: Improving Information Bottleneck with Structured Feature Learning (AAAI 2025 · similarity 0.86)
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.85)
- Decoupled Kullback-Leibler Divergence Loss (NeurIPS 2024 · similarity 0.85)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.85)
- A Rate-Distortion View of Uncertainty Quantification (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.