Research map / Learning theory and algorithms / Kernels and regression theory

On the Consistency of Kernel Methods with Dependent Observations

ICML 2024 · Learning theory and algorithms › Kernels and regression theory · approach: Kernel regression under shift · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces empirical weak convergence (EWC), a weak assumption on dependent data under which SVMs, kernel mean embeddings, and Hilbert-space-valued empirical expectations are shown to be consistent, including for infinite-dimensional outputs.

Problem

Consistency of kernel methods (SVMs, Gaussian processes, conditional kernel mean embeddings) is usually proven under i.i.d. or mixing data, yet these methods work well on non-i.i.d. dependent data such as dynamical systems.

Approach

Define EWC: a process is EWC if its empirical measure converges weakly to a (possibly random, path-dependent) limit measure, in probability or almost surely. Under EWC, prove L-consistency of regularized SVMs with a decreasing regularization sequence, using a separable Hilbert output space, which generalizes classical learning theory and covers conditional kernel mean embeddings.

Key results

• Establishes that SVMs with a continuously differentiable, convex, locally Lipschitz loss are L-consistent for EWC data with a suitable sequence of regularization parameters tending to zero.
• Shows that the MMD-based kernel mean embedding converges under EWC when the MMD metrizes weak convergence, and characterizes EWC for joint input-output processes.
• Shows that processes satisfying the weak law of large numbers for non-expectation (asymptotic mean stationary with LLNE) are EWC with the stationary mean as limit measure.

Surprising findings

• The process X_n = (-1)^n is EWC almost surely with limit measure (delta_-1 + delta_1)/2 despite not converging weakly.
• A weakly converging process (with a specific dyadic-block construction) is not EWC in probability, so weak convergence of marginals does not imply EWC.

Limitations

• Consistency is non-quantitative; no learning rates or convergence speed are provided.
• The weak convergence notion forces a continuously differentiable loss, which is more restrictive than the continuity required under total variation convergence.
• Requires separability of the space of continuous bounded functions, restricting to compact input and output sets in the main result.

Experimental setup

Builds on: support vector machines mixing processes ergodic theory statistical learning theory (Steinwart and Christmann)

Closest papers by meaning

• On Statistical Learning Theory for Distributional Inputs (ICML 2024 · similarity 0.87)
• Kernel conditional tests from learning-theoretic bounds (NeurIPS 2025 · similarity 0.87)
• Kernel Quantile Embeddings and Associated Probability Metrics (ICML 2025 · similarity 0.86)
• Learning to Embed Distributions via Maximum Kernel Entropy (NeurIPS 2024 · similarity 0.86)
• Generalization in Kernel Regression Under Realistic Assumptions (ICML 2024 · similarity 0.86)
• Gaussian Mixture Models in Hilbert Spaces via Kernel Methods (NeurIPS 2026 · similarity 0.85)