Research map / Large language models / Retrieval and knowledge
CoG-DQA: Chain-of-Guiding Learning with Large Language Models for Diagram Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoG-DQA, which uses GPT-3.5 multi-turn guiding prompts to steer frozen diagram parsing tools and then trains a T5-based answer module over visual and textual graphs for diagram question answering.
Problem
Diagram question answering needs domain knowledge and diagram parsing, but annotated data is scarce and LLM prompting is usually disconnected from visual parsing.
Approach
LLMs act as guiding chains over three stages (global knowledge, object extraction, relationship generation) that interact with frozen diagram parsing tools; the resulting visual and textual graphs feed a trained dual-graph GAT + cross-attention answer module initialized from Unified-QA T5.
Key results
- CoG-DQA reaches 78.85% on SQA-I (+5.32 over prior SOTA), 68.28% on CSDQA (+11.00), 54.60% on TQA-DMC (+1.27), and 79.14% on AI2D (+3.38).
- Under QCM->LE->A chain-of-thought setting on SQA-I, CoG-DQA reaches 89.32%, and 96.13% with ground-truth lecture and explanation.
- Adding CoG module as plug-in improves Unified-QA on CSDQA from 53.39 to 56.60 and MM-CoT on SQA-I from 73.53 to 75.68.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoG-DQA instead of MM-CoT-large* | DQA models · SQA-I test | accuracy: 73.53 -> 78.85 | Helped |
| Use CoG-DQA instead of MoCA | DQA models · TQA-DMC test | accuracy: 53.33 -> 54.60 | Helped |
| Use CoG-DQA instead of MM-CoT-large* | DQA models · CSDQA test | accuracy: 57.28 -> 68.28 | Helped |
| Use CoG-DQA instead of MM-CoT-large* | DQA models · AI2D test | accuracy: 75.76 -> 79.14 | Helped |
| Add CoG stage 1 (ablation) | Base T5-large · SQA-I test | accuracy: 73.13 -> 74.92; 1.79 ↑ | Helped |
| Add CoG stage 1 (ablation) | Base T5-large · CSDQA test | accuracy: 56.30 -> 58.85; 2.55 ↑ | Helped |
| Add CoG stage 2 to stage 1 (ablation) | CoG stage 1 · SQA-I test | accuracy: 74.92 -> 76.60; 3.47 ↑ | Helped |
| Add CoG stage 2 to stage 1 (ablation) | CoG stage 1 · CSDQA test | accuracy: 58.85 -> 59.87; 3.57 ↑ | Helped |
| Add the textual graph (TG) (ablation) | CoG stages 1 and 2 · SQA-I test | accuracy: 76.60 -> 77.00; 3.87 ↑ | Helped |
| Add the textual graph (TG) (ablation) | CoG stages 1 and 2 · CSDQA test | accuracy: 59.87 -> 62.49; 6.19 ↑ | Helped |
| Add CoG stage 3 and the visual graph (VG) (ablation) | CoG stages 1 and 2 with TG · SQA-I test | accuracy: 77.00 -> 77.14; 4.01 ↑ | Helped |
| Add CoG stage 3 and the visual graph (VG) (ablation) | CoG stages 1 and 2 with TG · CSDQA test | accuracy: 62.49 -> 65.36; 9.06 ↑ | Helped |
Surprising findings
- GPT-4 and LLaVA-1.5 prompt-based baselines underperform fine-tuned and QA-based models on the diagram datasets, suggesting weak transfer of general LLMs to DQA without diagram parsing interaction.
Limitations
- Reliance on GPT-3.5 prompting makes the pipeline dependent on a closed API and costly multi-turn queries.
- Uneven gains across datasets (+1.27% on TQA-DMC vs +11% on CSDQA).
Experimental setup
Datasets: SQA-I TQA-DMC CSDQA AI2D
Benchmarks: SQA-I TQA-DMC CSDQA AI2D
Metrics: accuracy prompt complete percentage
Baselines: MCAN BAN VisualBERT RAFR Unified-QA ISAAQ WSTQ MoCA GPT-3.5 GPT-4 LLaVA-1.5 MM-CoT
Builds on: Unified-QA MM-CoT ISAAQ CLIP SAM
Closest papers by meaning
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Multimodal Commonsense Knowledge Distillation for Visual Question Answering (Student Abstract) (AAAI 2025 · similarity 0.87)
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · similarity 0.87)
- Synthesize Step-by-Step: Tools Templates and LLMs as Data Generators for Reasoning-Based Chart VQA (CVPR 2024 · similarity 0.87)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.87)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.