Research map / Vision-language and multimodal / Multimodal LLMs

CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds CRIT, a graph-based, LLM-driven pipeline that generates cross-modal multi-hop QA with CoT traces over interleaved image-text content (natural images, video frames, scientific papers), and shows fine-tuning VLMs on it improves cross-modal reasoning benchmarks.

Problem

Existing multimodal benchmarks and interleaved training data rarely require complementary, multi-hop reasoning across text and images, so VLMs hallucinate and produce CoT traces poorly grounded in visual evidence.

Approach

Represent interleaved image-text content as a multimodal content graph (scene-graph entities from GQA, LLM-added text nodes), sample subgraphs with guaranteed cross-modal multi-hop chains, and use a text-only LLM to generate questions, CoT traces, and filter single-modality questions, avoiding VLMs in question generation.

Key results

• Qwen2.5-VL-7B fine-tuned on CRIT reaches 58.6 EM on the natural image split vs 28.3 for the base model and 35.1 for GPT-4o (CoT setting).
• Idefics2-8B trained on Mantis-Instruct+CRIT more than doubles SPIQA METEOR (3.60 to 10.53) and ROUGE-L (10.17 to 22.77).
• Augmenting CRIT to 210K samples with off-the-shelf-annotated images and videos raises Qwen2.5-VL-7B video-frame EM from 38.8 to 45.6.

Surprising findings

• Performance does not consistently decline with more hops, as additional hops mostly add textual reasoning that models handle well.
• Fine-tuning on CRIT improves several SPIQA metrics more than training on SPIQA itself does at matched size.

Limitations

• Dataset is entirely LLM-synthesized, so generator-specific biases and stylistic patterns may be present.
• Error accumulation across multi-stage LLM pipeline.
• Non-trivial computational overhead of the pipeline.

Experimental setup

Datasets: GQA ActivityNet Captions SPIQA Open Images Charades DiDeMo VidOR Mantis-Instruct

Benchmarks: CRIT SPIQA VEGA MMQA FCMR MuirBench BLINK MP-DocVQA MMStar MME SeedBench ChartQA

Metrics: EM F1 METEOR ROUGE-L CIDEr BERTScore F1 BLEU accuracy

Baselines: Gemini 2.0 Flash GPT-4o GPT-4o-mini Phi3.5-Vision LLaVA-Onevision Qwen2.5-VL Intern2.5-VL Idefics2 Mantis-Instruct

Builds on: GQA ActivityNet Captions SPIQA LoRA Mantis-Instruct

Closest papers by meaning

• ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.91)
• RBench-V: A Primary Assessment for Visual Reasoning Models with Multimodal Outputs (NeurIPS 2025 · similarity 0.90)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.90)
• Unveiling Fine-Grained Visual Traces: Evaluating MultiModal Interleaved Reasoning Chains in Multimodal STEM Tasks (NeurIPS 2026 · similarity 0.90)
• Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.90)
• CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models (AAAI 2025 · similarity 0.89)