Research map / Vision-language and multimodal / Multimodal LLMs

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AnyRef, an MLLM built on LLaVA-7B that performs pixel-level grounding and region-level referring expression generation from text, box, image, and audio references, using a refocusing mechanism that reweights the obj-token embedding with attention scores.

Problem

Existing MLLMs localize only at box level or accept only text referring instructions, lacking pixel-level perception and support for non-text references such as images and audio.

Approach

A unified referring representation maps text, box (RoIAlign), image (pooled CLIP features) and audio (ImageBind features) references into LLM token space, and a refocusing mechanism adds attention-weighted previous-token hidden states to the <obj> token embedding before SAM decoding, so mask losses implicitly supervise language generation.

Key results

• Referring expression segmentation: AnyRef (ft) reaches 76.9 cIoU on RefCOCO val, outperforming LISA-7B (ft) at 74.9 and most specialist models except UNINEXT.
• Audio-visual segmentation single-source: AnyRef achieves 82.8 mIoU versus 82.1 for AVSegformer, but lags AUSS on multi-source (55.6 vs 63.5 mIoU).
• Refocusing with lambda_f=0.1 raises RefCOCOg cIoU from 68.7 (no refocusing) to 70.0 and CIDEr from 71.1 to 73.7.

Measured effects reported in the paper

Surprising findings

• Without fine-tuning, the model's multi-source audio segmentation is weak, with AnyRef trailing AUSS and AVSegformer by large margins.
• Larger refocusing weights (1.0) hurt expression generation, and learnable lambda_f was sensitive to initialization.

Limitations

• Performance on multi-source audio segmentation is weak, struggling to cover more than one sounding object.
• Standard automatic metrics for referring expression generation poorly reflect quality of LLM-generated descriptions, requiring human evaluation.
• Lower segmentation performance than UNINEXT-H, which uses larger training data including video.

Experimental setup

Datasets: COCO-Stuff ADE20K PACO-LVIS RefClef RefCOCO RefCOCO+ RefCOCOg PhraseCut COCO PascalVOC Flickr30K Entities AVSBench

Benchmarks: RefCOCO RefCOCO+ RefCOCOg AVSBench COCO-20i PASCAL-5i Flickr30K Entities

Metrics: cIoU mIoU F-score one-shot mIoU few-shot mIoU Meteor CIDEr human preference accuracy

Baselines: LISA-7B UNINEXT PolyFormer SEEM SegGPT Painter AUSS AVSegFormer KOSMOS-2 Shikra IREG easyREG

Builds on: LLaVA LISA SAM ImageBind LoRA

Closest papers by meaning

• Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.93)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.91)
• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.91)
• MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.91)
• ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.90)
• Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.90)