Research map / Learning theory and algorithms / Kernels and regression theory

Bounds on $L_p$ Errors in Density Ratio Estimation via $f$-Divergence Loss Functions

ICLR 2025 · Learning theory and algorithms › Kernels and regression theory · approach: Statistical perspectives on generalization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives upper and lower bounds on L_p errors of density ratio estimators obtained by optimizing f-divergence variational loss functions, showing the lower bound has an exponential KL-divergence term that grows with p > 1.

Problem

Theoretical understanding of how accurately f-divergence loss-based density ratio estimation (DRE) can estimate the true density ratio in L_p error, including the role of KL divergence and dimensionality, is incomplete; minimax bounds do not reflect true estimation error.

Approach

Reformulates the sample-based f-divergence loss as a mu-representation loss, uses nearest-neighbor distance expectations and Lipschitz/bi-Lipschitz assumptions on the energy -log(dQ/dP) to derive f-divergence-independent upper and lower L_p error bounds, extended to early-stopped estimators via O_p(1/sqrt N) error margins.

Key results

• Lower bound contains exp((p-1)/p * KL(P||Q) - 1) term, so L_p error grows with KL for p>1 and faster as p increases.
• Numerical experiments on 5-dimensional synthetic data with KL from 1 to 14 show L_p errors for p>1 rising significantly with KL, while p=1 rises mildly.
• Experiments with dimensions 50, 100, 200 at KL=3 show errors increase with dimensionality for both alpha-divergence and KL losses.

Measured effects reported in the paper

Surprising findings

• The bounds are claimed to hold irrespective of the specific f-divergence loss, implying similar L_p error behavior across losses.

Limitations

• Bounds rely on strong assumptions such as Lipschitz/bi-Lipschitz continuity of the energy function and compact support.
• Analysis requires d >= 3 for the early-stopping error to be negligible.
• Experiments are limited to synthetic data.

Experimental setup

Datasets: synthetic datasets with controlled KL divergence and dimensionality

Benchmarks: synthetic DRE experiments

Metrics: L1 error L2 error L3 error

Baselines: KL-divergence loss alpha-divergence loss

Builds on: f-divergence variational representation minimax convergence rates for GAN/DRE

Closest papers by meaning

• Binary Losses for Density Ratio Estimation (ICLR 2025 · similarity 0.88)
• Density Ratio Estimation with Doubly Strong Robustness (ICML 2024 · similarity 0.87)
• Statistical and Geometrical properties of the Kernel Kullback-Leibler divergence (NeurIPS 2024 · similarity 0.85)
• Projection Pursuit Density Ratio Estimation (ICML 2025 · similarity 0.85)
• Loss Functions and Operators Generated by f-Divergences (ICML 2025 · similarity 0.84)
• Kernel conditional tests from learning-theoretic bounds (NeurIPS 2025 · similarity 0.84)