Research map / Point clouds / Point cloud classification
Floating-Point Neural Networks Can Represent Almost All Floating-Point Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that floating-point neural networks can represent almost all floating-point functions, giving necessary (distinguishability in the first layer) and sufficient conditions on activation functions, with corollaries for Sigmoid, tanh, ReLU, GELU, sin and others.
Problem
Expressive power of neural networks that use floating-point parameters and inexact floating-point arithmetic (round-off, overflow), which prior universal approximation results (real-valued parameters, exact operations) do not cover.
Approach
Identifies distinguishability of the input domain by the first layer as necessary for representing all floating-point functions, and proves it is sufficient under mild activation conditions (output zero and moderate values), exploiting round-off error and non-associativity of floating-point addition via "sequential addition" constructions.
Key results
- Shows that for floating-point networks to represent all floating-point functions, the first layer must distinguish inputs; cos-based correctly rounded networks fail on some functions over [-2^(p+7/2), 2^(p+7/2)] for float16.
- Theorem 3.4: under Condition 1 and distinguishability with range [-2^emax, 2^emax], a four-layer floating-point network represents all functions from X to F ∪ {±∞}.
- Corollary 3.8: correctly rounded Sigmoid and tanh networks represent all functions from F^d to F ∪ {±∞}; Corollary 3.11: Identity, ReLU, ELU, SeLU, GELU, Swish, Mish, sin cover (-2^(emax-2), 2^(emax-2))^d.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use correctly rounded Sigmoid or tanh activation | Floating-point neural networks · F^d | function representability | Helped |
| Use correctly rounded Identity, ReLU, ELU, SeLU, GELU, Swish, Mish, or sin activation | Floating-point neural networks · (-2^emax−2, 2^emax−2)^d_F | function representability | Helped |
| Require Condition 1 and σ-distinguishability with range [−2^emax, 2^emax]_F | Four-layer floating-point σ network · X ⊂ F^din | function representability | Helped |
| Use correctly rounded cos activation | Floating-point neural networks · [−2^⌊(p+7/2)⌋, 2^⌊(p+7/2)⌋]_F | function representability | Hurt |
Surprising findings
- Correctly rounded cos networks cannot represent some floating-point functions on a moderate domain, showing that not all practical activations suffice.
Limitations
- Does not establish minimum depth or width, or depth-width trade-offs.
- Domain is restricted for ReLU, GELU, etc. to magnitudes below about 1/8 of the largest float.
- Purely theoretical; no experiments.
Experimental setup
Baselines: Park et al. (2024) Hwang et al. (2024) Hwang et al. (2025) interval universal approximation
Builds on: Park et al. (2024) floating-point networks with ReLU/Step Hwang et al. (2024) fixed-point networks Ding et al. (2019) quantized ReLU networks Gonon et al. (2023) quantization expressivity
Closest papers by meaning
- On Minimum Depth and Width of Floating-Point Neural Networks for Representing Floating-Point Functions (ICML 2026 · similarity 0.94)
- Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients (ICML 2026 · similarity 0.94)
- On Expressive Power of Floating-Point Transformers (ICML 2026 · similarity 0.85)
- No Soundness in the Real World: On the Challenges of the Verification of Deployed Neural Networks (ICML 2025 · similarity 0.84)
- Graph Neural Networks and Arithmetic Circuits (NeurIPS 2024 · similarity 0.81)
- Complexity of Neural Network Training and ETR: Extensions with Effectively Continuous Functions (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.