Research map / Adversarial robustness and security / Certified and empirical robustness
Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes feature-space adversarial training for kernel regression, which has a closed-form inner maximization, an iterative kernel ridge regression solver, generalization bounds that adapt to noise without knowing it, and an extension to multiple kernel learning.
Problem
Adversarial training with input perturbations requires expensive min-max optimization, and its nonparametric (kernel) form is intractable; the paper targets efficient and analyzable adversarial training for kernel/nonparametric regression.
Approach
Shift adversarial perturbations from input space to RKHS feature space with an RKHS-norm ball; this gives a closed-form inner maximum (|y - f(x)| + δ||f||)^2, which is solved by an iterative reweighted kernel ridge regression via the eta-trick, and is shown to upper bound input-perturbed adversarial loss under kernel-distance balls.
Key results
- On Abalone with no attack, Adv Kern (delta ∝ n^-1/2) achieves R^2 0.57, matching Ridge CV at 0.57.
- Under l_inf test attack of radius 0.1, Adv Kern (delta ∝ n^-1/2) gets R^2 0.13 versus -0.16 for Ridge CV.
- Feature-space adversarial kernel training with RKHS radius 0.1 reaches R^2 0.30 under l_inf radius 0.1 attack, versus 0.13 for the default-radius variant and -0.11 to -0.14 for input-space adversarial training variants at that radius.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use feature-space adversarial training with δ∝n⁻¹ᐟ² instead of cross-validated ridge regression | Adversarial kernel training · Abalone, test-time ℓ₂ attack ‖Δx‖₂ ≤ 0.1 | R²: 0.26 (0.24–0.28) -> 0.39 (0.37–0.40) | Helped |
| Use feature-space adversarial training with δ∝n⁻¹ᐟ² instead of cross-validated ridge regression | Adversarial kernel training · Abalone, test-time ℓ∞ attack ‖Δx‖∞ ≤ 0.1 | R²: -0.16 (-0.20–-0.13) -> 0.13 (0.11–0.16) | Helped |
| Use feature-space adversarial training with δ∝n⁻¹ᐟ² instead of input-space adversarial training | Adversarial kernel training · Abalone, test-time ℓ∞ attack ‖Δx‖∞ ≤ 0.1 | R²: -0.14 (-0.17–-0.11) -> 0.13 (0.11–0.16) | Helped |
| Increase fixed feature-space radius from 0.01 to 0.1 (ablation) | Adversarial kernel training · Abalone, no attack | R²: 0.56 (0.55–0.58) -> 0.38 (0.37–0.39) | Hurt |
| Increase fixed feature-space radius from 0.01 to 0.1 (ablation) | Adversarial kernel training · Abalone, test-time ℓ∞ attack ‖Δx‖∞ ≤ 0.1 | R²: 0.18 (0.16–0.20) -> 0.30 (0.29–0.31) | Helped |
| Increase fixed feature-space radius from 0.01 to 0.1 (ablation) | Adversarial kernel training · Abalone, test-time ℓ₂ attack ‖Δx‖₂ ≤ 0.1 | R²: 0.40 (0.39–0.42) -> 0.35 (0.34–0.36) | Hurt |
| Use fixed feature-space radius 0.01 rather than δ∝n⁻¹ᐟ² (ablation) | Adversarial kernel training · Abalone, test-time ℓ∞ attack ‖Δx‖∞ ≤ 0.1 | R²: 0.13 (0.11–0.16) -> 0.18 (0.16–0.20) | Helped |
| Use fixed feature-space radius 0.1 rather than δ∝n⁻¹ᐟ² (ablation) | Adversarial kernel training · Abalone, no attack | R²: 0.57 (0.56–0.58) -> 0.38 (0.37–0.39) | Hurt |
Surprising findings
- Feature-space adversarially trained models remain robust to input-space attacks, outperforming input-space adversarial training baselines even under the same test-time threat model.
Limitations
- Analysis restricted to squared loss and regression; generalization bounds only in fixed-design, well-specified setting; misspecified and random-design cases left as future work.
- Scalability to large datasets not explored; CG and Nyström extensions are outside current scope.
- Benchmark evaluation on regression datasets with limited reporting in the main text; the Abalone table is the main robustness evaluation.
Experimental setup
Datasets: Abalone
Benchmarks: Abalone
Metrics: R^2 test mean square error
Baselines: Ridge CV (kernel ridge regression with cross-validation) input-space adversarial training
Builds on: kernel ridge regression multiple kernel learning eta-trick adversarial training for linear models
Closest papers by meaning
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.91)
- f-Domain Adversarial Learning: Theory and Algorithms (ICML 2021 · similarity 0.90)
- Ensemble Adversarial Training: Attacks and Defenses (ICLR 2018 · similarity 0.90)
- Fast and Scalable Adversarial Training of Kernel SVM via Doubly Stochastic Gradients (AAAI 2021 · similarity 0.90)
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (ICLR 2025 · similarity 0.90)
- A Kernel Perspective for Regularizing Deep Neural Networks (ICML 2019 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.