Research map / Continual learning / Continual fine-tuning of LLMs

FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Mixture-of-experts for multimodal continual tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FLAME, a sparse mixture-of-experts with per-modality routers and a low-rank, cursor-based continual learning scheme for flexi-modal multitask healthcare prediction.

Problem

Multimodal models must jointly train on tasks with heterogeneous modality combinations and then absorb new tasks with unseen modality combinations after deployment without forgetting or retraining from scratch.

Approach

Assign each modality a dedicated sample-level router dispatching tokens from any task into a shared expert pool; for continual learning, keep the expert pool fixed-size by appending rank-truncated per-stage additive expert updates, reading them out via a task cursor, and expanding only lightweight task-specific routers and heads.

Key results

• FLAME matches single-task multimodal baselines on AUROC within about 0.005 on six of nine tasks, and exceeds them on AUPRC for imbalanced binary tasks such as 48-IHM, RISK and DIAG (e.g., DIAG AUPRC 0.660 vs 0.601 for FuseMoE).
• FLAME-CL keeps end-of-stage AUROC within 0.01 of first-introduction values across four task sequences, while storing 5-15x fewer encoder parameters than Simple FT, EWC and LoRA.
• All-nine-task joint training trails per-task models by under 0.02 AUROC.

Surprising findings

• Joint all-nine-task training is reported to trail per-task models only modestly, and LoRA is used only as a reference since its parameter count grows with the task stream.

Limitations

• Continual learning evaluated on healthcare datasets only, with a limited number of task sequences (four).
• Theoretical alignment results rely on a linear surrogate of the nonlinear expert.
• FLAME-CL reserves a fixed rank budget per stage, so capacity must be planned in advance.

Experimental setup

Datasets: MIMIC-IV MIMIC-CXR eICU EMBED ADNI

Benchmarks: 48-IHM LOS 25-PHENO MOR RAD BIRADS RISK DENSITY DIAG

Metrics: AUROC AUPRC

Baselines: HighMMT FuseMoE FlexMoE Simple FT EWC LoRA Lifelong-PT

Builds on: HighMMT FuseMoE Flex-MoE EWC LoRA GPM

Closest papers by meaning

• Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.88)
• Rosetta: Composable Native Multimodal Pretraining (NeurIPS 2026 · similarity 0.87)
• Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.87)
• A Practitioner's Guide to Real-World Continual Multimodal Pretraining (NeurIPS 2024 · similarity 0.86)
• $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.86)
• Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.86)