Research map / Large language models / Agents, code and math
CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLOVA, a closed-loop visual assistant where LLM-generated programs over visual tools are reflected on (global-local, multimodal) using human feedback, and then both visual tools (via prompt tuning with validation) and LLM demonstrations are updated.
Problem
Tool-usage visual assistants that compose frozen visual tools with LLMs cannot learn new knowledge from failures, limiting adaptation to new environments.
Approach
A closed-loop inference-reflection-learning framework: multimodal global-local reflection identifies which tool failed, data is collected via LLM inference, open-vocabulary datasets, or Internet search, and visual tools are updated by instance-wise visual prompt tuning with validation-based prompt discarding, while LLMs are updated by storing correct and incorrect in-context examples.
Key results
- CLOVA reaches 54.6 on GQA and 65.6 on NLVRv2 with GPT-3.5, improving over VISPROG (49.8, 60.8) by roughly 5 points.
- On manually collected image editing CLOVA scores 65.4 versus 40.2 for VISPROG, and on knowledge tagging F1 0.502 versus 0.393.
- With LLaMA2-7B, updating LLMs raises GQA from 39.2 to 56.8 and updating visual tools reaches 60.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLOVA instead of VISPROG | Visual assistant · Image editing | accuracy: 40.2 -> 65.4 | Helped |
| Use CLOVA instead of VISPROG | Visual assistant · Knowledge tagging | F1 score: 0.393 -> 0.502 | Helped |
| Use CLOVA instead of MMICL | Visual assistant · NLVRv2 | accuracy: 62.2 -> 65.6 | Helped |
| Use CLOVA instead of MMICL | Visual assistant · GQA | accuracy: 64.4 -> 54.6 | Hurt |
| Add local reflection (ablation) | CLOVA · GQA | accuracy: 52.0 -> 54.6 | Helped |
| Add global reflection (ablation) | CLOVA · GQA | accuracy: 53.6 -> 54.6 | Helped |
| Include multimodal intermediate results in reflection (ablation) | CLOVA · GQA | accuracy: 48.8 -> 54.6 | Helped |
| Add plan generation (ablation) | CLOVA · GQA | accuracy: 50.0 -> 54.6 | Helped |
| Include incorrect examples in LLM prompts (ablation) | CLOVA · GQA | accuracy: 46.1 -> 54.6 | Helped |
| Include correct examples in LLM prompts (ablation) | CLOVA · GQA | accuracy: 48.2 -> 54.6 | Helped |
| Validate LLM in-context examples (ablation) | CLOVA · GQA | accuracy: 44.2 -> 54.6 | Helped |
| Validate visual-tool prompts (ablation) | CLOVA · GQA | accuracy: 42.8 -> 54.6 | Helped |
Surprising findings
- Updating LLMs with in-context examples helps the open-source LLaMA2-7B far more than GPT-4 (GQA gain of 21 points vs. 4 points).
Limitations
- Assumes no selection or loop structures in programs.
- Assumes at most one tool needs updating per task.
- Image editing and tagging evaluations use small manually collected datasets.
Experimental setup
Datasets: GQA NLVRv2 LVIS
Benchmarks: GQA NLVRv2 image editing (manually collected) factual knowledge tagging (manually collected)
Metrics: accuracy F1 score
Baselines: Otter MMICL GPT4TOOLs Visual ChatGPT InternGPT HuggingGPT ViperGPT VISPROG
Builds on: VISPROG CoOp VPT AssistGPT OpenAGI
Closest papers by meaning
- CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering (CVPR 2025 · similarity 0.88)
- Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces (NeurIPS 2026 · similarity 0.88)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.88)
- AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition (CVPR 2026 · similarity 0.87)
- Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.86)
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.