Research map / Vision-language and multimodal / Multimodal LLMs

MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoCHA, a LLaVA-style vision-language model that fuses four vision encoders (CLIP, SigLIP, DINOv2, ConvNeXt) via sparse mixture-of-experts connectors and a hierarchical group attention module, trained on Phi2-2.7B and Vicuna-7B.

Problem

Existing VLLMs rely on a single vision encoder or scale up models and resolution, which is costly and loses fine-grained visual detail, causing hallucination; the paper also seeks better integration of heterogeneous vision encoder features.

Approach

Replace the MLP connector with a Top-K sparse mixture-of-experts connector (MoEC) per vision encoder, concatenate encoder tokens along the sequence dimension, and apply hierarchical group attention (intra-group Top-M and inter-group Top-N selection with adaptive sigmoid gating) to reduce redundancy without extra trainable parameters.

Key results

• MoCHA (Vicuna-7B) reaches 89.98 POPE and 1744.09 MME, outperforming LLaVA-v1.5 (Vicuna-13B) by 4.08% on POPE and 2.99% on SQA-IMG.
• MoCHA (Phi2-2.7B) scores 89.95 POPE and 1701.60 MME, exceeding SPHINX-Plus (LLaMA2-13B) by 0.85% on POPE and 243.9 MME points.
• Ablation shows the full four-encoder MoEC+HGA configuration gives GQA 65.24, SQA-IMG 73.24, POPE 89.95, MMB 69.67 versus SigLIP-only 56.11/62.76/82.27/57.90.

Surprising findings

• Adding CLIP to SigLIP+ConvNeXt under a plain MLP connector degraded performance, which the authors attribute to feature redundancy between SigLIP and CLIP.

Limitations

• MoEC may suffer from knowledge entanglement and redundancy that hinders expert specialization.
• Some MoCHA numbers from GPT-API-based evaluation are averaged over only three runs.
• Gains in ablations are from a single training setup; a single vision encoder variant performs worse than the full model.

Experimental setup

Datasets: LAION-CC-SBU (558K subset) LLaVA-v1.5 665K instruction data

Benchmarks: GQA ScienceQA-IMG TextVQA MM-Vet POPE MME MMBench-EN MathVista

Metrics: accuracy MME perception+cognition score

Baselines: Qwen-VL-Chat LLaVA-v1.5 LLaVA-NeXT CuMo MoE-LLaVA LLaVA-Phi InternVL-Chat SPHINX-Plus VILA

Builds on: LLaVA-v1.5 MoE-LLaVA CuMo Cambrian-1

Closest papers by meaning

• Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design (CVPR 2026 · similarity 0.89)
• MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.89)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
• CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.88)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
• MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.88)