Research map / Vision-language and multimodal / Multimodal LLMs
Re-Aligning Language to Visual Objects with an Agentic Workflow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Real-LOD, an LLM-controlled agentic workflow (planning, VLM/LLM tool use, reflection) that re-aligns VLM-generated expressions to objects, builds the 0.18M-image Real-Data set, and trains a Swin-B LOD detector (Real-Model) that beats prior LOD methods on OmniLabel, DOD, RefCOCO and OVDEval.
Problem
VLM hallucinations in automatically generated object expressions (wrong name, color, shape, etc.) misalign language and visual objects, degrading language-based object detection training when data is scaled up.
Approach
An LLM agent (fine-tuned ChatGLM-6B) plans among five predefined states/actions, calls a VLM with cropped, extended-cropped, or highlighted image views and text prompts, and uses an LLM reflector for feedback in a cycle, so that low-SigLIP-score expressions are re-aligned instead of discarded.
Key results
- Real-Model reaches 36.5 AP-des on OmniLabel ALL vs 22.3 for FIBER (best prior), and 40.5 AP-des on OI vs 23.2 for mm-GDINO.
- Real-Agent planning achieves 74.7% re-alignment success rate vs 35.6% for random action selection.
- Ablation on COCO subset: AP-des rises from 21.2 (raw) to 22.2 (SigLIP-filtered) to 24.2 (with Real-LOD re-aligned data).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Real-LOD re-aligned expression data added back to SigLIP-filtered pairs (config C vs B) (ablation) | Real-Model Swin-B trained on Real-Data subset · OmniLabel COCO / O365 / OI | AP-des: COCO 22.2->24.2, O365 28.5->32.4, OI 31.4->33.5 | Helped |
| SigLIP filtering of mismatched raw expressions (B vs A) (ablation) | Real-Model training data · OmniLabel COCO / O365 / OI | AP-des: COCO 21.2->22.2, O365 27.6->28.5, OI 30.5->31.4 | Helped |
| Planning by LLM agent replaced with random state/action selection (ablation) | Real-LOD workflow · expression re-alignment success rate | success rate: 74.7% -> 35.6% | Hurt |
| Real-LOD agent planning vs random selection (ablation) | SigLIP matching score of refined expressions · sampled 11k expressions | avg SigLIP score improvement: 0.0673->0.1119 (66.27%) vs 0.0893 (32.69%) | Helped |
| Training Swin-B Real-Model on Real-Data (0.18M images) instead of prior LOD training sources | Real-Model · OmniLabel ALL / DOD / OVDEval | AP-des / Full / Avg AP: OmniLabel ALL 36.5 vs mm-GDINO 20.8; DOD 34.1 vs 24.2 | Helped |
| Applying Real-Data to other LOD models (MM-GDINO Swin-T and UNINEXT ResNet-50) (ablation) | MM-GDINO Swin-T; UNINEXT ResNet-50 · OmniLabel ALL AP-des and DOD Full | AP-des / Full: MM-GDINO 17.0->29.9 (OmniLabel), 23.0->30.8 (DOD); UNINEXT 6.0->23.4, 10.7->24.2 | Helped |
| Real-Data training with a larger GLIP (Swin-L, 17.5M images) comparison | Real-Model vs GLIP · OmniLabel ALL | AP-des | No clear effect |
Surprising findings
- A small Real-Data set (0.18M images) lets Real-Model outperform GLIP trained on 17.5M images, which the authors attribute to data quality.
Limitations
- Workflow is costly per expression and relies on repeated VLM/LLM calls.
- Random-selection and VLM-based reflection rely on LLM judgments that are not independently verified beyond SigLIP score and manual checks.
Experimental setup
Datasets: Objects365 OpenImage LVIS COCO RefCOCO/g/+ OmniLabel DOD OVDEval
Benchmarks: OmniLabel DOD RefCOCO RefCOCO+ RefCOCOg OVDEval
Metrics: AP-des AP-des-pos AP-des-S/M/L Full/Presence/Absence accuracy AP success rate SigLIP matching score
Baselines: MDETR GLIP mm-GDINO FIBER OWL-V2 UNINEXT GDINO OFA-DOD APE OmDet
Builds on: HuggingGPT Self-Refine Reflexion CRITIC LoRA LLaVA
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.89)
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.88)
- Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow (CVPR 2026 · similarity 0.88)
- GCD: Advancing Vision-Language Models for Incremental Object Detection via Global Alignment and Correspondence Distillation (AAAI 2025 · similarity 0.88)
- LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.