Research map / Learning theory and algorithms / Kernels and regression theory
Deep Principal Support Vector Machines for Nonlinear Sufficient Dimension Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Deep Principal SVM (DPSVM), a neural-network generalization of kernel principal SVM for nonlinear sufficient dimension reduction via a classification-ensemble objective, with unbiasedness and convergence-rate guarantees.
Problem
Nonlinear sufficient dimension reduction: finding functions of predictors X that capture all information about response Y, robust to outliers and scalable to large samples and complex inputs such as images.
Approach
Replaces the RKHS function class in kernel principal SVM with a deep ReLU network class and optimizes a hinge-loss SVM-style objective over binarized slices of the response (LVR/OVA), then aggregates slice-wise functions via eigendecomposition of a kernel-like matrix; proves unbiasedness w.r.t. the central sigma-field and a near-minimax rate n^{-2β/(p+2β)} up to log n.
Key results
- On synthetic settings, DPSVM achieves the best distance correlation in seven of nine settings (e.g., 0.80 in A-I vs 0.77 for KPSVM).
- On MNIST with nine estimated dimensions, DPSVM reaches 0.9862 accuracy versus 0.9883 for DDR and 0.9899 for raw LeNet.
- Theory: the estimator's L2 error rate is O(n^{-2β/(p+2β)}) up to a log n factor, matching the Stone minimax rate up to log n.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DPSVM instead of KPSVM | Nonlinear SDR · A-I | distance correlation: 0.77(0.01) -> 0.80(0.01) | Helped |
| use DPSVM instead of GSIR | Nonlinear SDR · A-II | distance correlation: 0.73(0.01) -> 0.74(0.02) | Helped |
| use DPSVM instead of KPSVM | Nonlinear SDR · A-III | distance correlation: 0.72(0.03) -> 0.77(0.02) | Helped |
| use DPSVM instead of DSRL | Nonlinear SDR · B-I | distance correlation: 0.68(0.17) -> 0.75(0.05) | Helped |
| use DPSVM instead of KPSVM | Nonlinear SDR · B-II | distance correlation: 0.59(0.10) -> 0.71(0.05) | Helped |
| use DPSVM instead of DSRL | Nonlinear SDR · B-III | distance correlation: 0.87(0.02) -> 0.75(0.05) | Hurt |
| use DPSVM instead of GMDDNet | Nonlinear SDR · C-I | distance correlation: 0.79(0.02) -> 0.76(0.04) | Hurt |
| use DPSVM instead of DSRL | Nonlinear SDR · C-II | distance correlation: 0.78(0.14) -> 0.78(0.04) | No clear effect |
| use DPSVM instead of DSRL | Nonlinear SDR · C-III | distance correlation: 0.89(0.07) -> 0.81(0.03) | Hurt |
| use DPSVM with LeNet instead of DDR with LeNet | MNIST classification · MNIST test | classification accuracy: 0.9883 -> 0.9862 | Hurt |
Surprising findings
- On setting B-III, DSRL (0.87) outperforms DPSVM (0.75), and on C-I GMDDNet (0.79) beats DPSVM (0.76).
Limitations
- Real-valued response must be discretized into binary slices, requiring multiple split schemes and fitting many networks, making it slow.
- Extending discretization to multivariate response is challenging since quantiles are hard to define.
Experimental setup
Datasets: MNIST Communities and Crime (CRIME) Connectionist Bench Sonar (SONAR) Optical Recognition of Handwritten Digits (OPTDIGITS) Breast Cancer Wisconsin (WDBC) synthetic models A, B, C
Benchmarks: MNIST CRIME SONAR OPTDIGITS WDBC synthetic settings A-I to C-III
Metrics: empirical distance correlation classification accuracy mean squared error
Baselines: KPSVM GSIR KSIR DDR DSRL GMDDNet
Builds on: principal support vector machines kernel principal SVM generalized sliced inverse regression ladle method
Closest papers by meaning
- Neural Networks Perform Sufficient Dimension Reduction (AAAI 2025 · similarity 0.86)
- High-Dimensional Analysis for Generalized Nonlinear Regression: From Asymptotics to Algorithm (AAAI 2024 · similarity 0.84)
- Enhancing Sufficient Dimension Reduction via Hellinger Correlation (ICML 2024 · similarity 0.83)
- Joker: Joint Optimization Framework for Lightweight Kernel Machines (ICML 2025 · similarity 0.83)
- The optimality of kernel classifiers in Sobolev space (ICLR 2024 · similarity 0.83)
- Faster Algorithms for Structured Linear and Kernel Support Vector Machines (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.