Research map / Probabilistic and scientific modeling / Molecules and proteins
FAME: Adaptive Functional Attention with Expert Routing for Function-on-Function Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FAME, an end-to-end function-on-function regression model combining bidirectional neural CDEs with a mixture-of-experts vector field, multi-head cross attention across input functions, and a neural CDE decoder, evaluated on synthetic GP data and three real datasets.
Problem
Function-on-function regression with irregularly and non-uniformly sampled input and output functions, where existing methods rely on fixed bases, kernels, or discretized grids and handle feature heterogeneity poorly.
Approach
Continuous attention is built from a bidirectional NCDE whose vector field is a router-weighted mixture of K experts, followed by cross attention across functions and an NCDE decoder that can be evaluated at arbitrary query points, giving sampling-invariant, Lipschitz-stable operator learning.
Key results
- On synthetic Case 1 with 200 samples, FAME achieves test MSE 0.0783 versus 0.1728 for the best kernel method and 0.2083 for FNN.
- On Case 7 with d=10 input functions, FAME reaches MSE 0.3330 versus 0.4498 for the best baseline (Gaussian process).
- On real-world datasets FAME gives the lowest MSE on all six tasks, e.g., Human3.6M Eating 0.0034 versus 0.0075 for Gaussian process, but the margin is smaller than on synthetic data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add bidirectional attention (ablation) | FAME · Synthetic Case 1, N=200 | MSE: 0.0812 -> 0.0783 | Helped |
| add MoE (ablation) | FAME · Synthetic Case 1, N=200 | MSE: 0.0828 -> 0.0783 | Helped |
| add cross-attention (ablation) | FAME · Synthetic Case 1, N=200 | MSE: 0.0815 -> 0.0783 | Helped |
| add MoE routing under feature heterogeneity (ablation) | FAME · Synthetic Case 4 (feature heterogeneity) | MSE: 0.0842 -> 0.0798 | Helped |
| increase expert count from 1 toward 3–5 (ablation) | FAME · Synthetic Case 8, d=10, heterogeneous features | MSE | Helped |
| increase expert count beyond 3–5 (ablation) | FAME · Synthetic Case 8, d=10, heterogeneous features | MSE | No clear effect |
Surprising findings
- The FAME ablation without MoE is only slightly worse on several settings, and FAME w/o Bi-dir is better than full FAME on Case 2 (0.1530 vs 0.1532) at 100 samples.
Limitations
- When the underlying operator varies little across the domain, simpler linear or low-rank models may be preferable.
- Training is substantially slower than classical baselines (about 207 s per epoch in the largest setting).
- Real-world datasets are small (e.g., 48 ETT samples), and the performance advantage is reduced there.
Experimental setup
Datasets: Gaussian process synthetic functions Hawaii Ocean Human3.6M ETT-small
Benchmarks: Gaussian process synthetic benchmark (Cases 1-8) Hawaii Ocean Human3.6M ETT-small
Metrics: mean squared error training time inference time peak memory
Baselines: B-spline basis expansion Fourier basis expansion FPCA Kernel Method (operator-valued kernel) Gaussian Process FNN
Builds on: neural controlled differential equations Transformer multi-head attention mixture of experts
Closest papers by meaning
- Ordinary Least Squares as an Attention Mechanism (NeurIPS 2026 · similarity 0.83)
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.83)
- Function Vectors in Large Language Models (ICLR 2024 · similarity 0.82)
- Neural Statistical Functions (NeurIPS 2026 · similarity 0.82)
- Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning (ICML 2025 · similarity 0.82)
- In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.