Research map / Vision-language and multimodal / Multimodal LLMs

ChatterBox: Multimodal Referring and Grounding with Chain-of-Questions

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the MCQ task (multimodal referring and grounding over chains of questions), the CB-300K benchmark built from Visual Genome with GPT-4 and rule filtering, and ChatterBox, a modular LLaVA-based model with a referent feedback mechanism.

Problem

Multimodal dialogues with referring and grounding often suffer referential confusion across continuous question chains, and existing MLLMs and benchmarks do not evaluate logically linked multi-round instance-level dialogue.

Approach

A modular design where the LLM emits a single grounding query token to an independent DINO-based grounding decoder and a RoIAlign referring module, plus a referent feedback mechanism that feeds the previous round's grounded box back into the next question as a token to resolve references.

Key results

• ChatterBox achieves an overall MCQ score T of 0.532 versus 0.421 for Kosmos-2 and 0.415 for Shikra on CB-300K.
• On COCO 2017 single-round grounding, ChatterBox reaches mIoU 0.710 and success rate 0.762, above Kosmos-2 (0.627, 0.688) and Shikra (0.606, 0.498).
• Training on CB-300K improves round-3 BERT score from 0.895 to 0.908 and T from 0.478 to 0.529 in ablation.

Measured effects reported in the paper

Surprising findings

• Using GPT-4 as an evaluator was avoided because of unstable randomness and preference bias, so a RoBERTa-large BERTScore plus IoU metric was used instead.

Limitations

• CB-CoQ subset is relatively small (about 8K threads) because strict rules and GPT-4 limits constrained generation.
• Dataset is built from Visual Genome annotations and GPT-generated Q&A, so it is synthetic and may carry GPT-4 biases.
• The referent feedback mechanism is only applied when grounding confidence exceeds 0.8 and the next question uses referential words.

Experimental setup

Datasets: CB-300K Visual Genome Objects365 RefCOCO RefCOCO+ RefCOCOg COCO 2017

Benchmarks: CB-300K RefCOCO RefCOCO+ RefCOCOg COCO 2017

Metrics: BERTScore IoU truncated thread score T METEOR mIoU success rate

Baselines: LLaVA GPT4RoI Kosmos-2 Shikra LISA

Builds on: LLaVA DINO GPT4RoI SAM iTPN

Closest papers by meaning

• F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.88)
• Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models (ICLR 2025 · similarity 0.87)
• MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.87)
• MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.87)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.86)
• Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.86)