Research map / Large language models / Retrieval and knowledge

Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models

ICLR 2025 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Chain-of-Action (CoA), a prompting framework that decomposes questions into action chains with plug-and-play web, knowledge, and data actions, verified by a multi-reference faith score (MRFS), and evaluates it on QA benchmarks and a Web3 QA deployment.

Problem

LLM question answering suffers from unfaithful hallucination inconsistent with real-time or domain facts, weak reasoning over compositional information, and high interaction and token cost when using retrieval.

Approach

Prompt the LLM to build an action chain of sub-questions with missing flags and guess answers, execute configurable retrieval actions (web, domain knowledge vectors, tabular data) per node, and correct guess answers via a ROUGE-inspired faith score; the model answers directly when its parametric knowledge suffices, reducing LLM calls.

Key results

• CoA with MRFS verification achieves the best accuracy in 12 of 14 benchmark scenarios, e.g., 70.7 on WebQA and 69.7 on QReCC versus SearchChain's 65.3 and 57.3.
• CoA reduces LLM interactions to about 4.0-4.4 per question versus 5.0-6.7 for Self-Ask, ReAct, and SearchChain.
• Expert evaluation in the Web3 case study gives CoA an overall score of 2.6 (full version) versus 1.9-2.0 for ReAct, Self-Ask, and DSP.

Measured effects reported in the paper

Surprising findings

• Adding retrieval does not always help: CoA without actions scores higher than several retrieval-based baselines on some tasks, and ReAct produced no result on DATE due to unresponsiveness.

Limitations

• Only text and tabular modalities implemented; multimodal (vision) support is future work.
• Evaluation relies on GPT-4 judge and a small expert panel of 20 for the Web3 case study, with human annotation on only 200 questions.
• The MRFS formula weights alpha, beta, gamma are not specified, and the threshold T is not reported.

Experimental setup

Datasets: WebQA DATE General Knowledge Social QA Truth QA StrategyQA FEVER ASQA QReCC

Benchmarks: WebQA DATE General Knowledge Social QA Truth QA StrategyQA FEVER ASQA QReCC

Metrics: GPT-EM ROUGE-L token usage latency number of LLM interactions expert rating (coverage, non-redundancy, readability, overal

Baselines: Few-shot prompting CoT Self-Consistency Tree of Thoughts Least-to-Most Auto-CoT ToolFormer Self-Ask ReAct DSP SearchChain

Builds on: chain-of-thought ReAct Self-Ask SearchChain DSP

Closest papers by meaning

• MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.89)
• MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.88)
• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.87)
• ChatterBox: Multimodal Referring and Grounding with Chain-of-Questions (AAAI 2025 · similarity 0.87)
• Tree-of-Reasoning Question Decomposition for Complex Question Answering with Large Language Models (AAAI 2024 · similarity 0.87)
• CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.87)