Research map / Optimization / Implicit bias and generalization
First-order ANIL provably learns representations despite overparametrisation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that first-order ANIL with a linear two-layer network provably learns a shared linear low-dimensional representation under infinite tasks, even when overparametrized, and unlearns the orthogonal complement.
Problem
Whether model-agnostic meta-learning (first-order ANIL) pretraining learns shared representations under architectural misspecification (network width larger than the true hidden dimension) in the few-shot regime with finite samples per task.
Approach
Analyzes FO-ANIL in the infinite-tasks limit with finite samples per task on a linear shared-representation model with an overparametrized two-layer linear network; proves the iterates asymptotically span the true column space, unlearn its orthogonal complement, and yield a single-step-adaptable initialization, without initial alignment.
Key results
- In the synthetic setting (d=50, k=5, m_test=20), FO-ANIL with 1 gradient step achieves excess risk 0.81 and with ridge 0.73, versus single-task ridge 1.84 and oracle ridge 0.50.
- FO-ANIL with infinite samples does not unlearn the orthogonal complement; its ridge excess risk is 1.04 at m_test=20 versus 0.73 for finite-sample FO-ANIL.
- Theorem shows asymptotic convergence: B_perp^T B_t -> 0, B_t w_t -> 0, and B_star^T B_t B_t^T B_star -> Lambda_star for any finite number of samples per task under infinite tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use finite rather than infinite samples per training task (ablation) | FO-ANIL · New-task test tasks, mtest=20, 1-GD | excess risk: 1.78 \pm 0.02 -> 0.81 \pm 0.01 | Helped |
| Use finite rather than infinite samples per training task (ablation) | FO-ANIL · New-task test tasks, mtest=20, Ridge | excess risk: 1.04 \pm 0.02 -> 0.73 \pm 0.03 | Helped |
| Use finite rather than infinite samples per training task (ablation) | FO-ANIL · New-task test tasks, mtest=30, 1-GD | excess risk: 1.19 \pm 0.01 -> 0.64 \pm 0.01 | Helped |
| Use finite rather than infinite samples per training task (ablation) | FO-ANIL · New-task test tasks, mtest=30, Ridge | excess risk: 0.84 \pm 0.02 -> 0.57 \pm 0.02 | Helped |
| Replace FO-ANIL with FO-MAML | FO-ANIL · New-task test tasks, mtest=20, 1-GD | excess risk: 0.81 \pm 0.01 -> 0.81 \pm 0.01 | No clear effect |
| Replace FO-ANIL with FO-MAML | FO-ANIL · New-task test tasks, mtest=30, 1-GD | excess risk: 0.64 \pm 0.01 -> 0.63 \pm 0.01 | No clear effect |
| Use a learned representation with Burer-Monteiro rather than single-task ridge | Burer-Monteiro · New-task test tasks, mtest=20 | excess risk: 1.84 \pm 0.03 -> 1.23\pm 0.03 | Helped |
| Use a learned representation with Burer-Monteiro rather than single-task ridge | Burer-Monteiro · New-task test tasks, mtest=30 | excess risk: 1.63 \pm 0.02 -> 1.03\pm 0.02 | Helped |
Surprising findings
- Finite samples per task are required for FO-ANIL to unlearn the orthogonal complement; with infinite samples the orthogonal complement is not unlearned, hurting test performance.
- Overparametrization causes a slowdown: the unlearning rate of the orthogonal complement is polynomial (O(1/t)) rather than exponential as in the well-specified case.
Limitations
- Analysis assumes infinite number of tasks, zero-mean task parameters, and identity task covariance.
- Features assumed standard Gaussian; extension to general distributions open.
- Results only asymptotic; no convergence rate for the feature-learning limit.
Experimental setup
Datasets: synthetic linear regression tasks
Benchmarks: synthetic linear regression tasks (excess risk on 1000 test tasks)
Metrics: excess risk squared singular values of B_star^T B_t and B_perp^T B_t
Baselines: Single-task ridge Oracle ridge Burer-Monteiro factorization FO-MAML FO-ANIL infinite samples
Builds on: MAML ANIL FO-MAML Burer-Monteiro factorization
Closest papers by meaning
- MAML and ANIL Provably Learn Representations (ICML 2022 · similarity 0.90)
- Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters (NeurIPS 2020 · similarity 0.89)
- Statistically and Computationally Efficient Linear Meta-representation Learning (NeurIPS 2021 · similarity 0.88)
- Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (ICLR 2018 · similarity 0.88)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (ICML 2017 · similarity 0.88)
- Provable Meta-Learning with Low-Rank Adaptations (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.