Research map / Architectures and efficiency / RNNs and normalization
Position: Categorical Deep Learning is an Algebraic Theory of All Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper proposing Categorical Deep Learning: using monads, their algebras, and lax algebras over the 2-category of parametric maps (Para) as a unified algebraic theory that recovers geometric deep learning constraints and implementations such as RNNs, recursive nets, and weight tying.
Problem
Lack of a general-purpose framework that bridges top-down specification of architectures via constraints (e.g., equivariance from geometric deep learning) and bottom-up specification via implementations, and that also covers computer-science constructs like lists, trees, and automata.
Approach
Model neural network layers as monad algebra homomorphisms (generalizing equivariance), with endofunctor (co)algebras for data structures and automata, and extend to lax algebras of free parametric monads in the 2-category Para to capture parameters, weight tying, and recurrent/recursive architectures from first principles.
Key results
- Shows that equivariant layers are monad algebra homomorphisms for group action monads, recovering GDL examples such as graph permutation equivariance, spherical CNNs, and G-CNNs.
- Shows that folds over lists and trees and unfolds for Mealy machines are endofunctor (co)algebra homomorphisms, generalising equivariance to recursive and recurrent structures.
- Argues that lax algebras of free parametric monads derive recurrent and recursive networks, with the lax cells being comonoids that model weight tying (no empirical evaluation).
Limitations
- Purely theoretical and position-based; no empirical experiments validate the framework's utility for designing or training architectures.
- Many claims (e.g., RNN and recursive network derivations via lax algebras) are only sketched in appendices due to space constraints.
- Restricts the main GDL recovery to linear equivariant layers; nonlinearities and parametric monads are only treated in later sections.
Experimental setup
Builds on: geometric deep learning Para 2-category algebraically free monads Kelly's unified transfinite construction categorical gradient-based learning
Closest papers by meaning
- Position: Algebra Unveils Deep Learning - An Invitation to Neuroalgebraic Geometry (ICML 2025 · similarity 0.84)
- Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory (ICLR 2025 · similarity 0.83)
- Transport of Algebraic Structure to Latent Embeddings (ICML 2024 · similarity 0.83)
- When Representations Align: Universality in Representation Learning Dynamics (ICML 2024 · similarity 0.83)
- Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions (ICML 2024 · similarity 0.82)
- A Theory of Formalisms for Representing Knowledge (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.