Research map / Label-efficient and robust learning / Recognition and long-tail
Disentangling Representations through Multi-task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and experimentally that optimal multi-task evidence-accumulation classifiers (RNNs, LSTMs, GPT-2) learn disentangled, abstract latent representations of noisy latent ground truth, enabling zero-shot OOD generalization.
Problem
Under what conditions do agents learn disentangled, abstract representations (world models) of latent factors from noisy, non-linearly observed evidence, and why do they generalize zero-shot OOD?
Approach
Any optimal multi-task binary classifier implicitly encodes the MLE of the ground-truth latent via its estimated distances to classification boundaries; with N_task >= D random linear boundaries and observation noise, a linear (or affine) readout of the latent state recovers the latent, with closed-form decoding and orthogonality emerging as N_task grows.
Key results
- Median OOD r^2 of 0.96 and ID r^2 of 0.97 across 5 example RNNs for N_task=24, D=2 (linear decoder trained on 3 of 4 quadrants).
- OOD generalization is poor when N_task < D and rises when N_task >= D; transformers reach near-perfect performance whenever N_task >= D.
- Maximum RNN OOD r^2 in free RT task matches the theoretical optimal-estimator bound r^2 = 1 - 0.48/T.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of simultaneous classification tasks | RNN multi-task learning · OOD latent-factor prediction | r² | Helped |
| Use GPT-2 transformers instead of RNNs | Multi-task latent-factor prediction · OOD latent-factor prediction | r² | Helped |
| Increase input noise during RNN pretraining | RNN multi-task learning · OOD latent-factor prediction, tested at σ=0.2 | r² | Helped |
| Remove input noise (ablation) | RNN multi-task learning · Continuous-attractor formation | continuous attractor formation | Hurt |
| Replace ReLU encoder nonlinearities with a quadratic nonlinearity (ablation) | RNN multi-task learning · OOD latent-factor prediction | r² | No clear effect |
| Double the noise standard deviation in one dimension (ablation) | RNN multi-task learning · OOD latent-factor prediction, N_task=24, D=2 | r²: median r^2 = 0.96 | No clear effect |
| Use autocorrelated AR(1) noise with coefficient 0.7 (ablation) | RNN multi-task learning · OOD latent-factor prediction, N_task=24, D=2 | r²: median r^2 = 0.95 | Hurt |
| Use perfectly correlated latent inputs (ρ=1) (ablation) | RNN multi-task learning · OOD latent-factor prediction | generalization performance | Hurt |
| Use Gumbel noise instead of Gaussian noise (ablation) | RNN multi-task learning · OOD latent-factor prediction | r²: median r^2 = 0.95 from 0.96 | Hurt |
| Train on interleaved linear and multiplicative classification tasks (ablation) | RNN multi-task learning · OOD latent-factor prediction | r²: median r^2=0.94,0.97 respectively | Helped |
| Train the free-reaction-time task rather than the fixed-reaction-time task (ablation) | RNN multi-task learning · OOD latent-factor prediction with a small number of tasks | r² | Helped |
| Change the fixed-reaction-time objective and loss to MSE with symmetric labels and tanh output (ablation) | RNN multi-task learning · OOD latent-factor prediction | generalization performance | No clear effect |
Surprising findings
- Noise in the inputs is necessary: networks trained without input noise do not form a 2D continuous attractor, and increasing training noise improves OOD generalization despite lower pretraining accuracy.
- Context-dependent (one task at a time) RNNs collapse the latent space into separate line attractors per task and cannot decode the latent factors.
Limitations
- Theory requires linear (affine-in-latent) decision boundaries; extension to general manifold boundaries left open.
- Experiments are on synthetic low-dimensional tasks (D=2 mainly) rather than real-world data.
Experimental setup
Datasets: synthetic multi-task evidence accumulation data (sampled ground truth x* in [-0.5,0.5]^D with Gaussian noise and random
Benchmarks: synthetic multi-task evidence accumulation tasks
Metrics: OOD r^2 ID r^2 angles between latent decoders sparsity eigenvalues of linearized dynamics
Baselines: Context-dependent RNN Vanilla RNN LSTM Leaky RNN
Builds on: Disentangled representations theory Leaky RNN / context-dependent computation Platonic representation hypothesis
Closest papers by meaning
- Leveraging sparse and shared feature activations for disentangled representation learning (NeurIPS 2023 · similarity 0.90)
- A Framework for the Quantitative Evaluation of Disentangled Representations (ICLR 2018 · similarity 0.90)
- Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning (AAAI 2024 · similarity 0.89)
- Disentangled Representation Learning in Non-Markovian Causal Systems (NeurIPS 2024 · similarity 0.88)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.87)
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.