Research map / Architectures and efficiency / RNNs and normalization

Approximation to Smooth Functions by Low-Rank Swish Networks

ICML 2025 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves that low-rank Swish networks (narrow bias-free linear layers between nonlinear Swish layers) can approximate any Hölder-ball function to arbitrary error, with linear-layer width at most one-third of nonlinear width, and runs small UCI experiments.

Problem

Low-rank compression is widely used and hardware-friendly, but lacks theoretical justification from universal approximation theory; universal approximation results for Swish networks are also scarce.

Approach

Insert a sufficiently narrow bias-free linear layer between each pair of adjacent Swish layers, and build a constructive approximation of Hölder functions via sum-product of Taylor polynomials and approximate bump functions, showing the linear width can be at most one-third of the nonlinear width with constant depth.

Key results

• Theorem 4.1 gives approximation error bound for Hölder ball functions with linear hidden layer width at most one-third of nonlinear width for smooth cases (kappa >= 2).
• Corollary 4.2: for any eps, a constant-depth low-rank Swish network approximates f to eps with width O(eps^{-d/beta}) and nonzero parameters O(eps^{-d/beta}).
• On 8 UCI datasets, low-rank and classical Swish networks show no significant difference in 7 of 8 datasets; on BikeSharing the low-rank network has significantly lower RMSE (0.070 vs 0.100, t=3.90).

Measured effects reported in the paper

Surprising findings

• On BikeSharing the low-rank network significantly outperforms the classical network in RMSE.

Limitations

• Results are existence/approximation-theoretic; no training or optimization guarantees.
• Experiments are small-scale UCI tabular tasks and do not test large models.
• Suffers from curse of dimensionality in the general Hölder case.

Experimental setup

Datasets: Iris Rice BankMarketing Adult RealEstate Abalone WineQuality BikeSharing

Benchmarks: UCI classification datasets (Iris, Rice, BankMarketing, Adult) UCI regression datasets (RealEstate, Abalone, WineQuality, BikeSharing)

Metrics: accuracy RMSE paired t-statistic

Baselines: classical feedforward Swish network Ohn & Kim 2019

Builds on: Yarotsky 2017 ReLU approximation Ohn & Kim 2019 Gühring & Raslan 2021 low-rank compression (Denil et al. 2013)

Closest papers by meaning

• Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (CVPR 2020 · similarity 0.88)
• COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation (NeurIPS 2025 · similarity 0.87)
• ReLU Network with Width $d+\mathcal{O}(1)$ Can Achieve Optimal Approximation Rate (ICML 2024 · similarity 0.86)
• Characterizing ResNet's Universal Approximation Capability (ICML 2024 · similarity 0.86)
• Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks (NeurIPS 2025 · similarity 0.86)
• A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.86)