Research map / Continual learning / Continual fine-tuning of LLMs

Rosetta: Composable Native Multimodal Pretraining

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Mixture-of-experts for multimodal continual tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Rosetta, a modular native multimodal pretraining architecture (unified attention, modality-specific FFN experts plus a global shared expert) with Momentum-Anchored Orthogonal Projection (MAOP) to reduce catastrophic forgetting when adding visual generation to language and understanding.

Problem

Adding continuous diffusion-based image generation to a language model trained with discrete next-token prediction causes gradient conflicts and catastrophic forgetting in MoE and MoT architectures.

Approach

Decouple the FFN into modality-specific routed experts plus a global shared expert, keep attention fully shared, and project incoming generative gradients orthogonally away from the optimizer's momentum (used as an anchor for prior knowledge), at zero extra memory cost.

Key results

• Rosetta reaches MMLU 49.2 and BBH 46.8 after full pretraining, while MoE and MoT drop to 26.3/27.1 MMLU and 0 BBH and MBPP.
• On visual generation, Rosetta achieves FID 14.05 vs 15.58 (MoT) and 17.80 (MoE), with T2I-CompBench color 52.9 vs 50.7 (MoT).
• Rosetta improves MMBench-EN to 52.5 and MMMU to 34.6 over its Pre-T2I checkpoint, while baselines degrade; MAOP adds no peak memory (58.8 GB per GPU) or throughput cost.

Surprising findings

• MoT's understanding routing also shifts under generative training, showing structural isolation alone does not prevent interference.
• Rosetta's MBPP score rises from 36.2 to 42.4 after adding T2I training.

Limitations

• Experiments are limited to a small 0.6B-based model upcycled to ~0.97B active parameters.
• Evaluations are on raw pretrained checkpoints without instruction tuning, so downstream usefulness is not established.
• Baselines are compared at matched active parameters but Rosetta has fewer total experts for language than MoE/MoT, which lowers its pre-generation MMLU.

Experimental setup

Datasets: MMU training samples T2I training samples text pretraining corpus (~300B tokens)

Benchmarks: MMLU BBH ARC-c MBPP MMMU MMBench-EN MMBench-CN POPE AI2D RealWorldQA COCO-30K T2I-CompBench

Metrics: accuracy FID CLIPScore HPSv2 T2I-CompBench attribute scores

Baselines: Standard MoE MoT (Bagel)

Builds on: Transfusion Mixture-of-Transformers PCGrad Symbiotic-MoE sparse upcycling

Closest papers by meaning

• FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning (NeurIPS 2026 · similarity 0.87)
• NExT-GPT: Any-to-Any Multimodal LLM (ICML 2024 · similarity 0.87)
• On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.87)
• Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models (NeurIPS 2025 · similarity 0.86)
• LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
• Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View (AAAI 2026 · similarity 0.86)