Research map / Architectures and efficiency / RNNs and normalization

Composing Linear Layers from Irreducibles

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Symmetry and equivariance in loss landscapes

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Expresses linear layers as compositions of rotor sandwich products built from bivectors, using a differentiable closed-form invariant decomposition, and replaces LLM attention key/query/value projections with O(log^2 d) parameter rotor layers that match block-Hadamard and low-rank baselines.

Problem

Whether linear layers in large models can be synthesized from a minimal set of geometric primitives with far fewer parameters than dense matrices, while retaining performance.

Approach

Represent linear maps as sums of rotor sandwich products in Clifford algebra Cl(n), parametrize rotors by bivectors via exponential map, and decompose general bivectors into commuting simple bivectors with a differentiable Clifford power-iteration, yielding closed-form differentiable rotors with O(log^2 d) parameters.</givenIdea>

Key results

• Rotor layers replacing one attention layer in LLaMa-3.2 1B reach 2.629 Wikitext2 log-PPL vs 2.636 for BH1 (second best), and use <=896 parameters for the query projection vs 4.19M dense.
• End-to-end rotor MLP on FMNIST reaches 88.36% vs 89.67% for dense.
• Rotor layers are slower in wall-clock time, roughly w x d times slower than dense in the authors' experiments.

Measured effects reported in the paper

Surprising findings

• Rotor layers match or beat LR4 and BH1 on several settings despite far fewer parameters.
• Rotor layers are slower in wall-clock runtime than dense layers despite fewer parameters.

Limitations

• Implementation uses dense matrix representations of rotors and does not yet exploit sparsity, so no wall-clock speedup.
• Evaluation is limited to layer-wise replacement in 1-1.5B models rather than full model conversion.
• No error bars are reported for accuracy results.

Experimental setup

Datasets: Wikitext2 C4 PTB Arc Challenge HellaSwag FMNIST

Benchmarks: Wikitext2 C4 PTB Arc Challenge HellaSwag FMNIST

Metrics: log perplexity accuracy parameter count FLOPs

Baselines: LR1 LR4 BH1 dense layers

Builds on: Clifford algebra Block-Hadamard DEQ LoRA

Closest papers by meaning

• Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices (NeurIPS 2024 · similarity 0.86)
• Transport of Algebraic Structure to Latent Embeddings (ICML 2024 · similarity 0.84)
• Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets (NeurIPS 2025 · similarity 0.83)
• Geometric Algebra Planes: Convex Implicit Neural Volumes (ICML 2025 · similarity 0.83)
• The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws (NeurIPS 2026 · similarity 0.83)
• Position: Algebra Unveils Deep Learning - An Invitation to Neuroalgebraic Geometry (ICML 2025 · similarity 0.83)