Research map / Vision-language and multimodal / Multimodal LLMs

DiVE-k: DIFFERENTIAL VISUAL REASONING FOR FINE-GRAINED IMAGE RECOGNITION

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiVE-k, which builds MCQs from a VLM's own top-k sampled category predictions and trains it with GRPO to select the correct option, improving zero-shot base-to-novel fine-grained recognition.

Problem

LVLMs underuse their text knowledge for fine-grained image recognition and RL with exact string-match rewards (as in ViRFT) is brittle, encourages memorization of training category names, and generalizes poorly to novel classes.

Approach

Use the base model's own top-k frequent predictions from K rollouts as multiple-choice options (ensuring the ground truth is included), then train with GRPO using a simple verifiable MCQ-correctness plus format reward, and use the same two-step (generate options, then select) pipeline at inference.

Key results

• Base-to-novel: DiVE-k reaches average HM 79.8 (base 80.8 / novel 78.8), +6.2 HM over ViRFT and +10.04 HM over Qwen2.5-VL-7B as stated in the abstract.
• Mixed-dataset base-to-novel: average HM 78.7, +9.0 over Qwen2.5-VL-7B and +4.0 over ViRFT.
• 4-shot: average accuracy 74.75, +7.73 over ViRFT and +10.85 over Qwen2.5-VL-7B.

Surprising findings

• The original ViRFT string-match reward contains a shortcut (partial substring match) that inflates training reward and evaluation accuracy; under LLM-based evaluation ViRFT drops sharply on CUB and Stanford Cars (e.g., 39.33 base on CUB).
• Standard SFT degrades sharply on novel classes (average novel accuracy drop of 33.5% relative to base model).

Limitations

• Two forward passes at inference roughly quintuple per-sample latency.
• Effectiveness depends on the base model already having high Pass@k accuracy; lower-capacity LVLMs may not benefit comparably.
• Slight regression on Oxford Pets versus ViRFT, attributed to more options causing more errors in step 2.

Experimental setup

Datasets: OxfordFlowers-102 CUB-200 OxfordPets-37 StanfordCars-196 FGVC Aircraft

Benchmarks: OxfordFlowers-102 CUB-200 OxfordPets-37 StanfordCars-196 FGVC Aircraft

Metrics: accuracy on base accuracy on novel harmonic mean

Baselines: Qwen2.5-VL-7B ViRFT SFT (full) SFT (LoRA) CLIP Gemini2.5-flash-lite GPT-5-mini Grok-4-fast Gemma3-12B

Builds on: ViRFT GRPO DeepSeek-R1 CLIP

Closest papers by meaning

• Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.89)
• Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)
• TTRV: Test-Time Reinforcement Learning for Vision Language Models (CVPR 2026 · similarity 0.89)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
• FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Latent Chain-of-Thought for Visual Reasoning (NeurIPS 2025 · similarity 0.88)