Research map / Learning theory and algorithms / Prediction and decision losses
Binary Losses for Density Ratio Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes all composite binary losses whose density-ratio estimators minimize a prescribed Bregman divergence, and derives convex losses (exponential weight, polynomial weights) that prioritize accurate estimation of large density ratios, improving importance-weighted parameter selection for deep domain adaptation.
Problem
Density ratio estimation via binary classification is implicitly tied to a Bregman divergence determined by the chosen loss; standard losses (logistic, boosting, square, KL-type) weight small density ratio values more, although large values often matter more in applications.
Approach
Starting from a prescribed Bregman generator phi and link g, it derives the unique form of strictly proper composite loss (and convex under a canonical choice g=(phi')^{-1}) that makes Algorithm 1 minimize that Bregman divergence; using increasing phi'' (e.g., exponential e^{2c}) yields losses emphasizing large density ratios.
Key results
- EW (exponential weight loss) has the highest mean accuracy under IWV on AmazonReviews (0.776 vs 0.773-0.774 for Boost, KuLSIF, LR) and under IWA (0.789 vs 0.766-0.781).
- EW is among the best methods in 56% of the 484 parameter selection problems, versus KuLSIF 23%, Boost 17%, LR 13%.
- In the Gaussian RKHS illustration, the exponential-weight estimator is flatter but avoids exploding estimates at small sample size where KuLSIF does; in covariate-shift regression the EW weight achieves squared L2(P) error below 0.040 versus above 0.067 for LR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EW rather than KuLSIF for importance-weighted validation | Deep domain adaptation parameter selection · AmazonReviews | accuracy: 0.774($\pm$.013) -> 0.776($\pm$.012) | Helped |
| Use EW rather than KuLSIF for importance-weighted aggregation | Deep domain adaptation parameter selection · AmazonReviews | accuracy: 0.781($\pm$.012) -> 0.789($\pm$.010) | Helped |
| Use EW rather than LR for importance-weighted validation | Deep domain adaptation parameter selection · HHAR | accuracy: 0.745($\pm$.040) -> 0.745($\pm$.038) | No clear effect |
| Use EW rather than KuLSIF for importance-weighted aggregation | Deep domain adaptation parameter selection · HHAR | accuracy: 0.717($\pm$.045) -> 0.722($\pm$.038) | Helped |
| Use EW rather than KuLSIF for importance-weighted validation | Deep domain adaptation parameter selection · MiniDomainNet | accuracy: 0.514($\pm$.020) -> 0.513($\pm$.018) | No clear effect |
| Use EW rather than KuLSIF for importance-weighted aggregation | Deep domain adaptation parameter selection · MiniDomainNet | accuracy: 0.521($\pm$.016) -> 0.522($\pm$.017) | Helped |
Limitations
- Sample complexity of the proposed losses is an open problem.
- The proposed loss functions are not self-concordant, so existing finite-sample rates for KuLSIF/self-concordant losses do not apply directly.
- Analysis is restricted to equal class priors (probability 1/2) in the main text.
Experimental setup
Datasets: AmazonReviews HHAR MiniDomainNet
Benchmarks: AmazonReviews HHAR MiniDomainNet
Metrics: accuracy
Baselines: KuLSIF LR (logistic loss) Boost (exponential loss) SO (source only) TB (target benchmark)
Builds on: KuLSIF logistic regression density ratio estimation Bregman divergence representation of density ratio estimation (Sugiyama et al.) importance weighted validation importance weighted aggregation
Closest papers by meaning
- Bounds on $L_p$ Errors in Density Ratio Estimation via $f$-Divergence Loss Functions (ICLR 2025 · similarity 0.88)
- Density Ratio Estimation with Conditional Probability Paths (ICML 2025 · similarity 0.86)
- Overcoming Saturation in Density Ratio Estimation by Iterated Regularization (ICML 2024 · similarity 0.85)
- Density Ratio Estimation-based Bayesian Optimization with Semi-Supervised Learning (ICML 2025 · similarity 0.85)
- Density Ratio Estimation with Doubly Strong Robustness (ICML 2024 · similarity 0.84)
- Classification Diffusion Models: Revitalizing Density Ratio Estimation (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.