Research map / Learning theory and algorithms / Kernels and regression theory
Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a Hilbert-space taxonomy of inductive transfer (convex-hull interpolation, linear-span extrapolation, out-of-span extrapolation) and a least-squares training scheme for function encoders, with a universal approximation result and comparisons on polynomial, CIFAR, 7-Scenes and MuJoCo benchmarks.
Problem
Transfer learning to new tasks without retraining is poorly characterized; it is unclear when and how learned models can transfer, especially to tasks outside the span of source tasks.
Approach
Model tasks as elements of a Hilbert space, define three geometric transfer types, and learn neural basis functions trained with a closed-form least-squares coefficient computation, so new tasks are represented by projection onto the learned basis at inference using a few samples.
Key results
- On the polynomial benchmark, the least-squares function encoder achieves low L2 error on type 2 and type 3 transfer where other baselines fail.
- On CIFAR, the function encoder performs best on type 1 and type 3 transfer, slightly ahead of Siamese networks.
- On MuJoCo Ant, the function encoder achieves the best overall transfer and is clearly best for type 2 transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use least-squares coefficient calculation instead of the inner-product method (ablation) | Function Encoder · Transfer benchmarks | performance | Helped |
| Use Function Encoder (LS) rather than the compared approaches | Function Encoder (LS) · Polynomial regression, type 1 transfer | L2 error | Mixed |
| Use Function Encoder (LS) rather than the compared approaches | Function Encoder (LS) · Polynomial regression, type 2 transfer | L2 error | Helped |
| Use Function Encoder (LS) rather than the compared approaches | Function Encoder (LS) · Polynomial regression, type 3 transfer | L2 error | Helped |
| Use Function Encoder rather than the compared approaches | Function Encoder · CIFAR-100, type 1 transfer | classification performance | Helped |
| Use Function Encoder rather than the compared approaches | Function Encoder · CIFAR-100, type 3 transfer | classification performance | Helped |
| Use Function Encoder rather than the compared approaches | Function Encoder · 7-Scenes, type 1 transfer | position estimation performance | Helped |
| Use Function Encoder rather than the compared approaches | Function Encoder · 7-Scenes, type 3 transfer | position estimation performance | Helped |
| Use Function Encoder rather than the compared approaches | Function Encoder · MuJoCo Ant, type 1 transfer | dynamics estimation performance | Mixed |
| Use Function Encoder rather than the compared approaches | Function Encoder · MuJoCo Ant, type 2 transfer | dynamics estimation performance | Helped |
| Use Function Encoder rather than the compared approaches | Function Encoder · MuJoCo Ant, type 3 transfer | dynamics estimation performance | Mixed |
Surprising findings
- Auto encoders briefly achieve strong type 3 transfer on MuJoCo, but this degrades as training minimizes training loss.
- Ad-hoc baselines (Siamese, prototypical) perform best on training tasks but drop sharply on type 1 or type 3 transfer.
Limitations
- Requires a well-defined inner product, which is hard to design for sequences and graphs.
- Does not address efficient representation of low-dimensional manifolds in infinite-dimensional spaces.
- Monte Carlo inner product induces function-specific weighting when input distributions shift, with unclear effects on performance.
Experimental setup
Datasets: polynomial regression (synthetic) CIFAR-100 7-Scenes hidden-parameter MuJoCo Ant
Benchmarks: polynomial regression CIFAR-100 few-shot classification 7-Scenes camera pose MuJoCo Ant dynamics
Metrics: L2 error classification accuracy position error dynamics prediction error
Baselines: function encoder (inner product) auto encoder transformer transformer functional encodings MAML Siamese networks prototypical networks brute force
Builds on: function encoder MAML transformer functional encodings residuals method
Closest papers by meaning
- On Hypothesis Transfer Learning of Functional Linear Models (ICML 2024 · similarity 0.86)
- Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context (ICML 2024 · similarity 0.85)
- Geometrically Aligned Transfer Encoder for Inductive Transfer in Regression Tasks (ICLR 2024 · similarity 0.85)
- In-context Learning on Function Classes Unveiled for Transformers (ICML 2024 · similarity 0.85)
- Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity (NeurIPS 2026 · similarity 0.85)
- Pretrained Transformer Efficiently Learns Low-Dimensional Target Functions In-Context (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.