Research map / Vision-language and multimodal / Human and embodied modeling
Task-aware Cross-modal Feature Refinement Transformer with Large Language Models for Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TCRT, a visual grounding transformer that uses a frozen Llama3.1-8B to produce task-aware priors, with textual conditional normalization in the visual backbone and cross-modal fusion for joint REC and RES.
Problem
End-to-end visual grounding struggles to precisely associate text descriptions with the correct visual regions, and LLMs trained only on text have a domain gap with visual inputs.
Approach
Adapt a frozen LLM to grounding via text-conditioned normalization of visual features and an object-text matching loss, then use the LLM's last-token task embeddings (box, segmentation, phrase) to guide cross-modal feature refinement and REC-RES interaction.
Key results
- TCRT reaches 91.07% val on RefCOCO for REC, +2.97 points over Vista-7B on RefCOCO val.
- On RES, TCRT improves over Vista-7B by +2.62, +2.12, +1.77 mIoU on RefCOCO splits.
- Ablation on RefCOCO val: removing both VA and VF drops accuracy from 91.07 to 87.62.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the LLM adaptation module (VA) (ablation) | TCRT · RefCOCO val | Prec@0.5: 87.62 -> 88.38 | Helped |
| Add the LLM adaptation module (VA) (ablation) | TCRT · RefCOCO testA | Prec@0.5: 88.60 -> 90.24 | Helped |
| Add the LLM adaptation module (VA) (ablation) | TCRT · RefCOCO testB | Prec@0.5: 84.04 -> 85.57 | Helped |
| Add task-aware cross-modal feature fusion (VF) (ablation) | TCRT · RefCOCO val | Prec@0.5: 88.38 -> 91.07 | Helped |
| Add task-aware cross-modal feature fusion (VF) (ablation) | TCRT · RefCOCO testA | Prec@0.5: 90.24 -> 92.85 | Helped |
| Add task-aware cross-modal feature fusion (VF) (ablation) | TCRT · RefCOCO testB | Prec@0.5: 85.57 -> 86.24 | Helped |
| Remove conditional normalization (ablation) | TCRT visual backbone · RefCOCOg test | localization accuracy | Hurt |
| Remove object-text matching loss (ablation) | TCRT · RefCOCO val | detection loss | Hurt |
Limitations
- Llama3.1-8B is frozen, so LLM-side adaptation relies on the adapter modules alone.
- Results are reported only on the four standard RefCOCO-family and ReferItGame benchmarks.
Experimental setup
Datasets: RefCOCO RefCOCOg RefCOCO+ ReferItGame
Benchmarks: RefCOCO RefCOCOg RefCOCO+ ReferItGame
Metrics: Prec@0.5 mIoU
Baselines: MAttNet CM-A-E Ref-NMS MCN RefTR SeqTR VG-LAW PVD Ferret-7B Vista-7B
Builds on: TransVG SeqTR VG-LAW RefTR Swin Transformer
Closest papers by meaning
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.89)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.89)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.89)
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.89)
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.89)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.