Research map / Label-efficient and robust learning / Recognition and long-tail
Deep Support Vectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeepKKT, a generalization of the KKT conditions to deep classifiers, to extract and synthesize Deep Support Vectors (DSVs) from a trained model, applied to few-shot dataset distillation, model inversion, and generation via label latents.
Problem
Deep learning needs massive data and offers no explicit decision criteria, unlike SVMs whose support vectors reconstruct the model and explain decisions. The paper asks how to recover SVM-like support vectors from trained deep networks.
Approach
Adapt the SVM KKT conditions (primal feasibility, dual feasibility, stationarity, plus a manifold condition) to multi-class deep models, using cross-entropy as a surrogate and optimizing input samples and Lagrange multipliers with the frozen model, so the resulting DSVs act as support vectors and can be used for distillation and label-conditioned generation.
Key results
- On CIFAR-10 with 1 img/class and 0.02% of real data, DSVs reach 22.69% accuracy versus 16.48% for DC, 15.41% for DSA, and 13.03% for DM.
- With 10 img/class (0.2% of data) DSVs reach 37.90% versus 29.77% for the best baseline (DM).
- In coreset selection on CIFAR-10 with 1 image per class, DSVs-selected samples reach 16.83% versus 14.26% for random selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DSVs instead of the strongest listed distillation baseline (DC) | CIFAR10 few-shot dataset distillation, 1 image/class and 1 shot/class · CIFAR10 | accuracy: 16.48$\pm$0.81 -> \textbf{22.69} $\pm$ 0.38 | Helped |
| Use DSVs instead of the strongest listed distillation baseline (DM) | CIFAR10 few-shot dataset distillation, 10 images/class and 10 shots/class · CIFAR10 | accuracy: 29.77$\pm$0.66 -> \textbf{37.90} $\pm$ 1.69 | Helped |
| Use DSVs instead of the strongest listed distillation baseline (DM) | CIFAR10 few-shot dataset distillation, 50 images/class and 50 shots/class · CIFAR10 | accuracy: 48.93$\pm$0.92 -> \textbf{53.56} $\pm$ 0.73 | Helped |
| Select coreset images by high DSV lambda values instead of random sampling | CIFAR10 coreset selection, 50 images/class (1%) · CIFAR10 | test accuracy: 46.16 $\pm$ 1.93 -> 48.91 $\pm$ 0.90 | Helped |
| Select coreset images by high DSV lambda values instead of random sampling | CIFAR10 coreset selection, 10 images/class (0.2%) · CIFAR10 | test accuracy: 30.08 $\pm$ 1.96 -> 33.69 $\pm$ 2.05 | Helped |
| Select coreset images by high DSV lambda values instead of random sampling | CIFAR10 coreset selection, 1 image/class (0.02%) · CIFAR10 | test accuracy: 14.26 $\pm$ 0.99 -> 16.83 $\pm$ 0.29 | Helped |
| Use only the primal loss, omitting stationarity and manifold loss terms (ablation) | DeepKKT DSV generation · DSV generation | sample meaningfulness | Hurt |
Surprising findings
- Synthesized DSVs can be generated with zero real images (initialized from other datasets), still giving 21.68% accuracy on CIFAR-10 in the 0-shot setting.
- Using soft label mixtures as latent variables produces semantically fused ImageNet images, so a classifier can act as a lightweight latent generative model.
Limitations
- The equivalence between DSVs and SVM support vectors is only argued intuitively, not proven rigorously.
- Complementary slackness is not explicitly enforced because it requires expensive function-space distances.
- Generation quality is evaluated mainly qualitatively; no standard generative metrics are reported.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN ImageNet
Benchmarks: CIFAR-10 few-shot dataset distillation CIFAR-10 coreset selection ImageNet DSV generation and latent interpolation
Metrics: test accuracy entropy Lagrange multiplier correlation with classwise accuracy
Baselines: DC DSA DM Random coreset selection
Builds on: KKT conditions SVM dataset distillation (DC/DSA/DM)
Closest papers by meaning
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.84)
- Are Large Kernels Better Teachers than Transformers for ConvNets? (ICML 2023 · similarity 0.84)
- Provable and Efficient Dataset Distillation for Kernel Ridge Regression (NeurIPS 2024 · similarity 0.83)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.82)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.82)
- On the expressiveness and spectral bias of KANs (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.