Research map / Vision-language and multimodal / Multimodal LLMs

Retrieving Counterfactuals Improves Visual In-Context Learning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CIRCLES, a visual in-context learning method that augments similarity-based demonstration retrieval with counterfactual examples retrieved via attribute-guided, training-free composed image retrieval, improving VLM accuracy on CUB, Flowers, OK-VQA and VizWiz.

Problem

Similarity-based demonstration retrieval for visual in-context learning picks correlated but non-causal examples, so VLMs learn spurious associations, especially with limited data or small models.

Approach

Use the VLM to extract decisive attributes and generate counterfactual captions by intervening on one attribute at a time, retrieve counterfactual-style images via composed image retrieval (OSrCIR) scored by image-caption and question-question similarity, and combine them with standard CLIP nearest neighbors in the prompt.

Key results

• On Gemma3-4B, CIRCLES reaches 71.97% accuracy on CUB vs 65.40% for RICES and 31.27 EM on OK-VQA vs 26.65 for RICES.
• On Qwen2.5-VL-3B, average EM across the four datasets is 71.44 for CIRCLES vs 69.67 for RICES.
• Under 75% training-data removal on CUB, CIRCLES's relative gain over RICES on Gemma3-4B grows from 10.05% to 16.28%.

Surprising findings

• Adding textual attributes without counterfactual retrieval (IR+Attr) slightly degrades performance on OK-VQA and VizWiz.
• CIR-only demonstrations perform far below IR-only on fine-grained classification (25.16% vs 65.40% CUB accuracy on Gemma3-4B).

Limitations

• Relies on VLM-extracted attributes, which can be non-discriminative, and on weak composed retrieval, causing failure cases.
• Gains on VQA benchmarks (OK-VQA, VizWiz) are modest; the method is not formal causal identification.
• Additional VLM calls for attribute extraction and counterfactual captioning add inference overhead.

Experimental setup

Datasets: CUB Flowers OK-VQA VizWiz ScienceQA

Benchmarks: CUB Flowers OK-VQA VizWiz ScienceQA

Metrics: accuracy weighted F1 exact match word-level F1

Baselines: None (zero-shot) Random RICES MUIER MMICES CIReVL RICES*

Builds on: RICES MUIER MMICES OSrCIR CLIP

Closest papers by meaning

• In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.88)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.87)
• ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.87)
• Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data (ICLR 2025 · similarity 0.87)
• Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning (NeurIPS 2025 · similarity 0.87)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.86)