Research map / Label-efficient and robust learning / Recognition and long-tail
Task structure and nonlinearity jointly determine learned representational geometry
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that in single-hidden-layer networks, Tanh learns representations aligned with target output geometry while ReLU retains more input geometry, explained by the asymmetric asymptotic saturation of ReLU causing neuron specialization.
Problem
How the input geometry, output (label) geometry, and nonlinearity jointly shape learned neural representational geometry, and in particular how disentangled and target-aligned learned representations are.
Approach
Parameterized classification tasks with controllable input-output kernel alignment, combined with learning-dynamics analysis of hidden-neuron input weights, show that Tanh's symmetric saturation drives neurons toward inter-class (label) axes while ReLU's one-sided saturation induces specialization to input regions, retaining input structure.
Key results
- Tanh networks learn representations with higher target alignment, parallelism score and CCGP than ReLU networks across tasks, while ReLU retains more input structure.
- For the delta-separable XOR task, Tanh networks discard input geometry for any delta above a small threshold, whereas ReLU alignment increases gradually with delta.
- A symmetric-saturating nonlinearity (clipped ReLU, min(max(x,0),1)) behaves almost identically to Tanh, indicating asymptotic symmetry drives target alignment more than behavior near the origin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ReLU with Tanh | single-hidden-layer networks · Unstructured four-cluster classification | target alignment | Helped |
| Use ReLU instead of Tanh | single-hidden-layer networks · Unstructured four-cluster classification | input alignment | Helped |
| Increase input-output alignment by increasing class separability | Tanh networks · δ-separable XOR classification | target alignment | Mixed |
| Increase input-output alignment by increasing class separability | ReLU networks · δ-separable XOR classification | target alignment | Helped |
| Increase training noise | Tanh networks · δ-separable classification | target alignment | Hurt |
| Increase training noise to modest levels | ReLU networks · δ-separable classification | target alignment | Helped |
| Replace ReLU with Tanh | deep networks · Synthetic multi-layer tasks | final-layer target alignment | Helped |
| Replace ReLU with Tanh | convolutional networks · CIFAR-10 and STL-10 | target alignment | Helped |
| Use a nonlinearity with two-sided saturation instead of ReLU (ablation) | single-hidden-layer networks · Random classification tasks with P=32 inputs | target alignment | Helped |
| Shift ReLU so its linear region contains the origin (ablation) | single-hidden-layer networks · Random classification tasks with P=32 inputs | target alignment | Helped |
| Allow output readout weights to be trained instead of freezing them (ablation) | single-hidden-layer networks · Representational geometry during training | target alignment | No clear effect |
| Use multidimensional outputs instead of a single output (ablation) | Tanh networks · Four-cluster, two-target classification | target alignment | Helped |
Surprising findings
- Learned representational geometry depends strongly on activation function choice even when task performance is high.
- Increasing input noise in Tanh networks shifts representations toward ReLU-like input-preserving geometry, and modest input noise raises target alignment in ReLU networks.
Limitations
- Main analysis uses single-hidden-layer networks with frozen, discretized output weights.
- Multi-layer and convolutional experiments are limited in scale and only qualitatively replicate the shallow results.
- Tasks use binary, symmetric label structure and clustered Gaussian inputs.
Experimental setup
Datasets: CIFAR-10 STL-10
Benchmarks: CIFAR-10 STL-10
Metrics: target alignment input alignment parallelism score cross-condition generalization performance linear decodability (SVM) test accuracy
Baselines: ReLU networks
Builds on: neural collapse cross-condition generalization performance parallelism score
Closest papers by meaning
- Parameter symmetries determine representational geometry in overparameterized nonlinear networks (NeurIPS 2026 · similarity 0.89)
- Feature Learning beyond the Lazy-Rich Dichotomy: Insights from Representational Geometry (ICML 2025 · similarity 0.89)
- How are linear representations learned? Exact solutions to the dynamics of abstraction (NeurIPS 2026 · similarity 0.88)
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.88)
- RNNs perform task computations by dynamically warping neural representations (NeurIPS 2025 · similarity 0.88)
- When Representations Align: Universality in Representation Learning Dynamics (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.