Research map / Learning theory and algorithms / Sample complexity
Overcoming Saturation in Density Ratio Estimation by Iterated Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that kernel density ratio estimators (KuLSIF, LR, Exp, SQ) saturate, and proposes iterated Tikhonov regularization that restores fast error rates, validated on synthetic density ratios and importance-weighted ensembling for unsupervised domain adaptation.
Problem
Kernel-based density ratio estimation methods suffer from error saturation, so they cannot reach fast convergence rates on highly regular density ratio problems.
Approach
Applies iterated Tikhonov regularization (each step regularizes toward the previous iterate) to a Bregman-divergence framework covering many loss functions for density ratio estimation, proving non-saturating rates and improving empirical performance for iteration counts up to 10.
Key results
- On the Geometric Figures benchmark, iterated KuLSIF, Exp and SQ improve over non-iterated versions; average error for KuLSIF drops from 11.205 to 11.198 and Exp from 18.775 to 12.890.
- In domain adaptation over Amazon Reviews, iterated KuLSIF averages 0.790 accuracy vs 0.787 for non-iterated KuLSIF.
- On breast cancer ablation, iterated KuLSIF reaches error 1.19 vs 1.21 for KuLSIF, and iterated Exp 3.17 vs 3.47.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add iterated regularization to Exp (ablation) | Exp density ratio estimation · Geometric Figures (Avg.) | twice Bregman divergence error: 18.775(\pm 0.123) -> 12.890(\pm 0.032) | Helped |
| Add iterated regularization to KuLSIF (ablation) | KuLSIF density ratio estimation · Geometric Figures (Avg.) | twice Bregman divergence error: 11.205(\pm 0.006) -> 11.198(\pm 0.008) | Helped |
| Add iterated regularization to SQ (ablation) | SQ density ratio estimation · Geometric Figures (Avg.) | twice Bregman divergence error: 11.810(\pm 0.015) -> 11.400(\pm 0.018) | Helped |
| Add iterated regularization to LR (ablation) | LR density ratio estimation · Geometric Figures (Avg.) | twice Bregman divergence error: 11.400(\pm 0.023) -> 11.400(\pm 0.021) | No clear effect |
| Add iterated regularization to density ratio estimators (ablation) | KuLSIF, Exp, LR, and SQ in domain adaptation · MiniDomainNet | target classification accuracy | Mixed |
| Add iterated regularization to density ratio estimators (ablation) | KuLSIF, Exp, LR, and SQ in domain adaptation · Amazon Reviews | target classification accuracy | Helped |
| Add iterated regularization to KuLSIF and Exp (ablation) | KuLSIF and Exp density ratio estimation in domain adaptation · HHAR | target classification accuracy | Helped |
| Add iterated regularization to LR and SQ (ablation) | LR and SQ density ratio estimation in domain adaptation · HHAR | target classification accuracy | No clear effect |
| Add iterated regularization to KuLSIF (ablation) | KuLSIF density ratio estimation · Amazon Reviews (Avg.) | target classification accuracy: 0.787 (\pm 0.011) -> 0.790 (\pm 0.010) | Helped |
Surprising findings
- Iterated LR showed no improvement over non-iterated LR on the geometric benchmark, though performance stayed the same.
- Exp degrades heavily with non-iterated regularization on some datasets (e.g., 70.827 error on c3,d1.57), which iteration largely fixes.
Limitations
- Iteration count t introduces a new hyperparameter; theory uses a priori parameter choice requiring knowledge of regularity index r.
- Constant in the theoretical bound can be large, so empirical benchmarks are needed for support.
- Theory does not apply to general deep learning due to additional regularizations.
Experimental setup
Datasets: Gaussian mixture density ratio datasets (synthetic) Breast cancer dataset Amazon Reviews MiniDomainNet HHAR
Benchmarks: Geometric Figures density ratio benchmark MiniDomainNet Amazon Reviews HHAR Breast cancer density ratio ablation
Metrics: twice Bregman divergence error L1 error of density ratio target classification accuracy
Baselines: KuLSIF LR Exp SQ DANN CDAN DSAN DIRT CoDATS MMDA
Builds on: KuLSIF Bregmanization iterated Tikhonov regularization importance-weighted ensembling (Gruber et al. / Zellinger-style aggregation)
Closest papers by meaning
- Density Ratio Estimation with Conditional Probability Paths (ICML 2025 · similarity 0.86)
- Binary Losses for Density Ratio Estimation (ICLR 2025 · similarity 0.85)
- SD-KDE: Score-Debiased Kernel Density Estimation (NeurIPS 2025 · similarity 0.84)
- Nonparametric Estimation of a Factorizable Density using Diffusion Models (NeurIPS 2026 · similarity 0.84)
- Revive Re-weighting in Imbalanced Learning by Density Ratio Estimation (NeurIPS 2024 · similarity 0.84)
- Projection Pursuit Density Ratio Estimation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.