Research map / Vision-language and multimodal / Cross-modal learning
Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a per-sample probabilistic expert-routing framework that selects among text/numeric modality paths and STL/MTL heads for depression and anxiety prediction, evaluated on synthetic data and a psychotherapy notes dataset.
Problem
Multimodal multitask prediction where modality informativeness and task correlation vary across samples; fixed fusion and globally shared multitask architectures are suboptimal, with psychotherapy outcome prediction (PHQ-9, GAD-7) as the motivating application.
Approach
A two-stage soft/hard router first picks among four modality paths (text-only, numeric-only, textualized-numeric+text, numeric+text-embedding) and then picks STL or MTL heads per sample, mixing the eight resulting experts trained end-to-end with heteroscedastic losses.
Key results
- Routing (soft, learned paths) achieves PHQ-9 RMSE 3.62 and GAD-7 RMSE 3.34, versus 3.66/3.42 for the best fixed path T2 and 3.85/3.52 for heteroscedastic MTL.
- Text-only T1 is the worst fixed path (PHQ-9 RMSE 4.60, GAD-7 4.10), while T2 textualized-numeric plus text is best among fixed paths.
- On synthetic data the router concentrates mass on N2/T2 paths and prunes T1/N1 routes, with the paper claiming lower RMSE than static baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use T2 (textualized numeric features fused with notes) instead of T1 (text-only) | Fixed-path STL models · Psychotherapy data, PHQ-9 | RMSE: 4.60 -> 3.66 | Helped |
| Use T2 (textualized numeric features fused with notes) instead of T1 (text-only) | Fixed-path STL models · Psychotherapy data, GAD-7 | RMSE: 4.10 -> 3.42 | Helped |
| Add heteroscedastic loss to shared-encoder MTL (ablation) | MTL (Shared encoder) · Psychotherapy data, PHQ-9 | RMSE: 4.12 -> 3.85 | Helped |
| Add heteroscedastic loss to shared-encoder MTL (ablation) | MTL (Shared encoder) · Psychotherapy data, GAD-7 | RMSE: 3.78 -> 3.52 | Helped |
| Use learned soft routing instead of independent STL | Psychotherapy prediction model · Psychotherapy data, PHQ-9 | RMSE: 4.28 -> 3.62 | Helped |
| Use learned soft routing instead of independent STL | Psychotherapy prediction model · Psychotherapy data, GAD-7 | RMSE: 3.95 -> 3.34 | Helped |
| Use T2 instead of N2 | Fixed-path STL models · Psychotherapy data, PHQ-9 | RMSE: 3.80 -> 3.66 | Helped |
| Use T2 instead of N2 | Fixed-path STL models · Psychotherapy data, GAD-7 | RMSE: 3.58 -> 3.42 | Helped |
Surprising findings
- Text-only routes performed worst on PHQ-9, and the router pruned T1 and N1 routes on synthetic data.
- The routing gains over the best fixed path are small (about 0.04 RMSE on PHQ-9 and 0.08 on GAD-7).
Limitations
- Real-world evaluation uses a hybrid corpus augmented with synthetically generated encounters, causing distribution drift.
- Synthetic benchmark uses idealized, noise-free-label distributions that may not reflect real notes or questionnaires.
- No reported variance, significance testing, or details of the real-data train/test split in the main results.
Experimental setup
Datasets: Synthetic multitask regression benchmark Mental Health in Healthcare Workers psychotherapy dataset (N=534, augmented with synthetic records)
Benchmarks: Synthetic multitask regression benchmark Mental Health in Healthcare Workers psychotherapy dataset
Metrics: RMSE
Baselines: T1 text-only N1 numerical-only T2 textualized numeric + text N2 numerical + embedded text STL independent MTL shared encoder Heteroscedastic MTL
Builds on: Mixture-of-Experts Multi-gate Mixture-of-Experts (MMoE) Gumbel-Softmax Sentence-BERT BERT
Closest papers by meaning
- FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning (NeurIPS 2026 · similarity 0.86)
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.83)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.83)
- MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series (NeurIPS 2025 · similarity 0.83)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.83)
- CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.