Research map / Vision-language and multimodal / Multimodal LLMs

ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ChartMoE, an InternLM-XComposer2-based MLLM whose MLP connector is replaced by a mixture-of-experts connector with experts initialized from chart-to-table/JSON/code alignment, plus a ~1M-sample ChartMoE-Align dataset.

Problem

Chart understanding MLLMs struggle to extract faithful data and give reliable analysis from charts; standard connector alignment (chart-to-table) erodes general capabilities and random or co-upcycled MoE initialization leads to poor convergence or expert homogenization.

Approach

Replace the linear connector with a MoE connector whose experts are each initialized from a different chart alignment task (table, JSON, code) plus the vanilla connector, then fine-tune with LoRA on the LLM; also drop the load-balancing loss since visual token distributions are naturally imbalanced.

Key results

• ChartMoE+PoT reaches 84.64% average relaxed accuracy at 5% margin on ChartQA, versus 80.48% for prior SOTA TinyChart+PoT.
• On ChartBench, ChartMoE achieves 51.67% ALL versus 48.41% for InternlmXC-v2 base; extra chart types improve from 39.72% to 55.58%.
• Ablation: diverse alignment initialization gives 78.76% vs 77.48% for co-upcycling and 73.20% for random init on ChartQA (Acc@0.05 average).

Measured effects reported in the paper

Surprising findings

• Chart-to-table/JSON/code alignment alone lowers ChartQA accuracy relative to the vanilla baseline before SFT, but improves after SFT on ChartQA.
• Removing the load-balancing (bz) loss improves performance, as the visual token distribution is naturally imbalanced across charts.

Limitations

• Requires chart table/JSON/code alignment tasks for initialization; non-chart multimodal tasks need new alignment designs.
• Not plug-and-play like LoRA; the router must be retrained when new experts are added.
• String-matching evaluation penalizes correct answers in ChartQA augmented split, so reported gains may partly reflect overfitting.

Experimental setup

Datasets: ChartQA PlotQA ChartY MMC-Instruction ChartGemma instructions LLaVA-CC3M MME

Benchmarks: ChartQA ChartBench ChartFC ChartCheck MME

Metrics: relaxed accuracy Acc+ accuracy

Baselines: InternlmXC-v2 TinyChart ChartAst ChartGemma ChartLlama OneChart ChartVLM Deplot + LLaVA-v1.6 DocOwl-v1.5 Qwen-VL LLaVA-v1.5 ChartInstruct

Builds on: InternLM-XComposer2 (InternlmXC-v2) LLaVA CuMo co-upcycling LoRA ChartReformer MMC

Closest papers by meaning

• ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.90)
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (NeurIPS 2024 · similarity 0.89)
• EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.88)
• ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding (NeurIPS 2025 · similarity 0.87)
• ChartE$^{3}$: A Comprehensive Benchmark for End-to-End Chart Editing (ICML 2026 · similarity 0.87)
• Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts (CVPR 2026 · similarity 0.86)