Research map / Vision-language and multimodal / Multimodal LLMs

Same or Not? Enhancing Visual Perception in Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TWIN, a 561K-query dataset asking whether two images show the same object instance, and FGVQA, a 12K-query fine-grained VQA benchmark suite; GRPO post-training of Qwen2.5-VL-3B and InternVL3.5-1B on TWIN improves fine-grained recognition, including on unseen domains.

Problem

Vision-language models remain coarse-grained and miss subtle instance-level visual details, partly because their training corpora emphasize category-level recognition and general reasoning rather than fine-grained discrimination.

Approach

Reframe fine-grained perception as a pairwise same-instance VQA task with human-verified hard negatives and DreamBooth-generated negatives, and train VLMs with GRPO using only binary answer-correctness rewards.

Key results

• Qwen2.5-VL-3B + TWIN improves mean FGVQA accuracy from 53.0% to 67.0%, with +19.3 points on Inquire (54.4% to 73.7%).
• RL outperforms SFT on a 136K subset: mean FGVQA 57.5% vs 53.8%, with CUB 65.0% vs 53.9%.
• Hard negatives add about 4.5 points on average (63.1% vs 58.6%) in the ablation on a 250K subset.

Surprising findings

• Training on TWIN with GRPO leaves text-only performance largely unchanged, with MMLU 64.7 to 64.6 and GSM8K dropping from 70.8 to 68.8 for Qwen2.5-VL.
• In-context learning with TWIN examples hurts Qwen2.5-VL-3B badly (39.5% mean), well below the zero-shot baseline of 53.0%.

Limitations

• Gemma3 outperforms the trained Qwen2.5-VL on MET and Landmarks, so gains are not uniform across domains.
• Failure cases remain on fine color differences, extreme viewpoint variation, and incomplete animal views.
• Only binary outcome rewards are used; the authors note more structured rewards are unexplored.

Experimental setup

Datasets: TWIN Amazon Reviews ILIAS Google Landmarks v2 MET CUB Inquire Pets SUN397 CIFAR100

Benchmarks: FGVQA (TWIN-Eval, ILIAS, Landmarks, MET, CUB, Inquire) RealWorldQA POPE CV-Bench NLVR2 AI2D SEED-Bench ChartQA TextVQA BLINK MMMU VLMsBias

Metrics: accuracy linear probe accuracy KNN accuracy

Baselines: GPT-4o Gemini 2.5 Flash PerceptionLM 3B Moondream 2 Gemma3 4B SmolVLM2 2.2B SFT on Gemini-filtered responses

Builds on: GRPO DreamBooth CLIP verl

Closest papers by meaning

• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.91)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
• The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.90)
• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.90)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)