Research map / Optimization / Implicit bias and generalization

Deep Sturm–Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions

ICML 2025 · Optimization › Implicit bias and generalization · approach: Explicit and implicit regularization interplay

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Deep Sturm–Liouville (DSL), a function approximator that builds orthogonal basis functions by solving learnable Sturm–Liouville problems along field lines of a neural vector field, enabling 1D regularization across the input space.

Problem

Neural networks generalize poorly because regularization is applied per sample (0D); the paper seeks continuous 1D regularization along trajectories spanning the input space.

Approach

Define a neural vector field whose field lines traverse the domain; along each line solve a learnable Sturm–Liouville eigenvalue problem to get orthogonal 1D basis functions, linearly combine them, and regularize via the number of basis functions and an eigenvalue penalty. Gradients are obtained via implicit differentiation through the shooting/boundary-finding procedure.

Key results

• DSL reaches 84.28% on Adult, 91.14% on Dry Bean, 83.10% on Bank Marketing, 97.93% on MNIST and 58.38% on CIFAR-10, comparable to NODE baselines (e.g., 91.45% Dry Bean, 96.8% MNIST, 58.9% CIFAR-10).
• In low-sample experiments on the Bank dataset, DSL consistently outperforms a standard neural network across training sizes 100 to 800.
• Accuracy on Dry Bean is stable across a wide range of basis sizes, peaking around 10 eigenfunctions.

Measured effects reported in the paper

Surprising findings

• The DSL ablation without the vector field performs slightly better than full DSL on Bank Marketing (84.63% vs 83.10%).

Limitations

• Gradient and prediction computation are expensive due to ODE solving and shooting/binary search.
• Training can be unstable; ODE solver may fail to detect boundary in rare configurations.
• Only comparable (not superior) accuracy to baselines on most benchmarks; Mnist/Cifar10 baseline numbers taken from prior work.

Experimental setup

Datasets: Adult Dry Bean Bank Marketing MNIST CIFAR-10

Benchmarks: Adult Dry Bean Bank Marketing MNIST CIFAR-10

Metrics: classification accuracy

Baselines: Neural ODE (NODE) standard neural network (MLP) DSL ablation without vector field

Builds on: Neural ODEs Sturm-Liouville theory Elliptic eigenvalue problem Prüfer substitution

Closest papers by meaning

• Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.85)
• Improving Generalization and Convergence by Enhancing Implicit Regularization (NeurIPS 2024 · similarity 0.84)
• Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.84)
• Mask in the Mirror: Implicit Sparsification (ICLR 2025 · similarity 0.83)
• Implicit regularization of deep residual networks towards neural ODEs (ICLR 2024 · similarity 0.83)
• Investigating the Overlooked Hessian Structure: From CNNs to LLMs (ICML 2025 · similarity 0.83)