Research map / Adversarial robustness and security / Certified and empirical robustness

Robust Feature Learning for Multi-Index Models in High Dimensions

ICLR 2025 · Adversarial robustness and security › Certified and empirical robustness · approach: Theory of adversarial risk · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that for multi-index targets with ℓ2-bounded adversaries and squared loss, the hidden directions give a Bayes-optimal robust projection, so robust learning reduces to standard feature learning plus robust training of a readout, with sample complexity independent of ambient dimension.

Problem

Whether neural networks can retain statistical adaptivity to low-dimensional (multi-index) structure when trained for adversarial robustness under ℓ2-bounded perturbations, i.e., whether robust learning is as statistically easy as standard learning in high dimensions.

Approach

Under the assumption that the index coordinates are independent of the remaining coordinates, the projection onto the hidden directions U is Bayes optimal for ℓ2 adversarial risk under squared loss. So one learns U by standard feature learning on the first layer, then adversarially trains only a second-layer readout on the low-dimensional representation.

Key results

• Theory: the projection onto the hidden directions U achieves adversarial risk at most AR* under l2 attacks with squared loss (Theorem 3.1), so robust learning needs only the standard feature subspace.
• Sample complexity of the second phase and neurons is independent of ambient dimension d given a feature learning oracle; e.g., total n = O~(d/eps~^2) for Gaussian single-index polynomials.
• Synthetic experiments (d=100, N=100): initializing first layer from standard training then adversarial training outperforms full adversarial training from random initialization.

Measured effects reported in the paper

Surprising findings

• Adversarially training the first layer may be suboptimal, since full adversarial training from random initialization can fail to achieve dimensionality reduction even with infinite samples.

Limitations

• Requires the index coordinates to be statistically independent of the remaining coordinates (Assumption on input distribution); shown necessary in a linear example.
• Focuses on l2 perturbations; l_infinity case left open.
• Analysis assumes an oracle for feature learning; computing the inner max in phase 2 is not analyzed.

Experimental setup

Datasets: synthetic Gaussian single-index data (ReLU, tanh, He2 teachers)

Benchmarks: synthetic single-index regression with adversarial test error

Metrics: adversarial risk (squared loss) adversarial test error

Baselines: Full adversarial training from random initialization (full AD) Standard training followed by adversarial training (SD + AD)

Builds on: Mean-field / gradient-based feature learning for single- and multi-index models Adversarial training (min-max robust optimization) Information exponent and generative exponent

Closest papers by meaning

• A Spectral View of Adversarially Robust Features (NeurIPS 2018 · similarity 0.88)
• Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.87)
• Adversarially Robust Generalization Requires More Data (NeurIPS 2018 · similarity 0.87)
• Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (ICML 2020 · similarity 0.86)
• Consistent Adversarially Robust Linear Classification: Non-Parametric Setting (ICML 2024 · similarity 0.85)
• Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.85)