Research map / Learning theory and algorithms / Sample complexity

Position: Algebra Unveils Deep Learning - An Invitation to Neuroalgebraic Geometry

ICML 2025 · Learning theory and algorithms › Sample complexity · approach: PAC learnability theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing that neural networks with polynomial (algebraic) activations have semi-algebraic neuromanifolds that can be studied with algebraic geometry, and outlining a dictionary linking invariants (dimension, degree, singularities, fibers, critical points) to ML aspects (sample complexity, expressivity, implicit bias, training dynamics).

Problem

Understanding the function spaces (neuromanifolds) parameterized by machine learning models and how their geometry relates to sample complexity, expressivity, implicit bias, identifiability and training dynamics, especially for algebraic models.

Approach

Frame neural networks with polynomial activations as semi-algebraic varieties and map algebro-geometric invariants (dimension, degree, covering number, singularities, parameterization fibers, Euclidean distance degree, data discriminants, dynamical invariants) onto machine learning questions, naming this research direction neuroalgebraic geometry.

Key results

• Presents a dictionary mapping ML aspects (sample complexity, expressivity, identifiability, implicit bias, optimization) to algebraic geometry invariants (dimension, degree, covering number, singularities, fibers, critical points).
• Worked toy example: two-layer linear network has a determinantal variety neuromanifold of dimension N1(N0+N2-N1), with spurious critical points for N0=N1=N2=2.
• Gives a covering-number-based sample complexity bound for ERM over compact neuromanifolds, logarithmic in the covering number.

Surprising findings

• For N0=N1=N2=2 linear two-layer networks, the parameter-space loss has spurious critical points forming a 4-dimensional component despite the function-space problem having a unique critical point.

Limitations

• Focuses on algebraic (polynomial) models, which are rarely used as activations in practice; extension to non-algebraic models is only approximate.
• Many connections are reviews of existing literature and conjectures rather than new theorems or experiments.

Experimental setup

Builds on: singular learning theory information geometry neural tangent kernel metric algebraic geometry algebraic statistics

Closest papers by meaning

• Transport of Algebraic Structure to Latent Embeddings (ICML 2024 · similarity 0.86)
• Semialgebraic Neural Networks: From roots to representations (ICLR 2025 · similarity 0.86)
• Position: Categorical Deep Learning is an Algebraic Theory of All Architectures (ICML 2024 · similarity 0.84)
• Lie Neurons: Adjoint-Equivariant Neural Networks for Semisimple Lie Algebras (ICML 2024 · similarity 0.84)
• Learning Polynomial Problems with $SL(2, \mathbb{R})$-Equivariance (ICLR 2024 · similarity 0.84)
• Approximating Latent Manifolds in Neural Networks via Vanishing Ideals (ICML 2025 · similarity 0.84)