Research map / Optimization / Implicit bias and generalization
Activation-Descent Regularization for Input Optimization of ReLU Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADR-GD, an input-optimization method for ReLU networks that relaxes activation patterns into sigmoid-surrogate variables and adds activation-descent regularization terms, optimized with perturbed gradient descent and primal-dual updates.
Problem
Gradient-based input optimization of ReLU networks is unreliable because gradients are local to the current activation polytope and ignore how the input crossing into new activation patterns changes the objective.
Approach
Augment inputs with continuous sigmoid-relaxed activation-pattern variables, derive a steepest activation-descent direction in the discrete pattern space, and add regularizers (objective term on activation variables plus a feasibility/constraint loss normalized by hyperplane norms) so input updates are aligned with activation-space descent; the Lagrangian is minimized with primal-dual descent and
Key results
- On MNIST, PGD attack success on the adversarially trained robust model drops to 4.92% while ADR-GD achieves 27.93%, and ADR-GD outperforms PGD and FGSM across MNIST, CIFAR10 and ImageNet.
- ADR-GD per-iteration runtime is roughly an order of magnitude slower than PGD because it optimizes additional activation variables.
- Enhanced DDPG reaches nearly 2,000 average episode reward in Ant, versus the reported ~1,000 ceiling for DDPG in prior work.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ADR-GD instead of PGD | MNIST robust model adversarial attack · MNIST robust model | attack success rate: 4.92% -> 27.93% | Helped |
| Use ADR-GD instead of PGD | Adversarial optimization on robust models · MNIST and CIFAR10 robust models | attack success rate | Helped |
| Use ADR-GD instead of PGD | Adversarial optimization on naturally trained models · MNIST, CIFAR10, and ImageNet naturally trained models | attack success rate | No clear effect |
| Use ADR-GD instead of PGD | Action refinement during DRL exploitation · MuJoCo environments | average episode reward | Helped |
| Use ADR-GD instead of PGD | Adversarial attacks on neural policies · FruitBot and CoinRun | average episode reward | Hurt |
| Add the objective gradient on activation variables η (ablation) | ADR-GD · Randomized neural networks | optimized objective value F(x*) | Helped |
| Add the constraint-loss gradient on input x (ablation) | ADR-GD · Randomized neural networks | optimized objective value F(x*) | Helped |
| Add normalization to the constraint loss (ablation) | ADR-GD · Randomized neural networks | optimized objective value F(x*) | No clear effect |
| Add perturbations to activation-pattern variables (ablation) | ADR-GD · Randomized neural networks | optimized objective value F(x*) | Helped |
| Use a sigmoid-surrogate slope α of 2000 (ablation) | ADR-GD · Randomized neural networks | optimized objective value | Mixed |
Surprising findings
- Normalizing the constraint loss by hyperplane norms is unnecessary for small networks (M3 performs comparably), though it matters for deeper networks.
Limitations
- Per-iteration cost is roughly an order of magnitude higher than PGD due to optimizing activation variables.
- Scalability to larger networks is limited; only the last layer of Wide ResNet-50-2 is optimized.
- No convergence or theoretical complexity analysis provided.
Experimental setup
Datasets: MNIST CIFAR10 ImageNet Procgen FruitBot Procgen CoinRun MuJoCo environments
Benchmarks: MNIST CIFAR10 ImageNet MuJoCo Procgen FruitBot Procgen CoinRun
Metrics: attack success rate optimized loss average episode reward optimized objective value F(x*) runtime per iteration
Baselines: FGSM PGD DDPG TD3 Perturbed GD
Builds on: Perturbed gradient descent FGSM PGD
Closest papers by meaning
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.89)
- Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time (ICML 2024 · similarity 0.87)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.87)
- Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification (NeurIPS 2025 · similarity 0.87)
- ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models (ICLR 2024 · similarity 0.87)
- ReLUs Are Sufficient for Learning Implicit Neural Representations (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.