Research map / Learning theory and algorithms / Prediction and decision losses

What does automatic differentiation compute for neural networks?

ICLR 2024 · Learning theory and algorithms › Prediction and decision losses · approach: Theory of neural network sensitivity

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretical investigation of what automatic differentiation (forward/reverse mode) actually computes for neural networks containing non-differentiable functions and control flow.

Problem

AD is guaranteed correct only for differentiable programs without control flow; for neural networks with non-differentiable points it may return arbitrary values, and it is unclear what it computes.

Approach

Analyzes AD output on neural networks with non-differentiable functions and control flows, characterizing what AD computes rather than assuming correctness; the abstract is truncated so the specific characterization is not stated.

Closest papers by meaning

• On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters (ICML 2023 · similarity 0.93)
• On Correctness of Automatic Differentiation for Non-Differentiable Functions (NeurIPS 2020 · similarity 0.89)
• Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operators (NeurIPS 2024 · similarity 0.85)
• Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.83)
• Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations (NeurIPS 2025 · similarity 0.83)
• Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context (ICML 2024 · similarity 0.83)