Research map / Vision-language and multimodal / Multimodal LLMs

Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-R1 trains Qwen2.5-VL models for fine-grained visual recognition via CoT SFT on 404 synthesized rationales followed by TAPO, an RL variant with intra-class and inter-class augmentation, using 4-shot data per category.

Problem

MLLMs underperform contrastive CLIP models on fine-grained visual recognition, require large annotated data to adapt, and overfit to seen sub-categories while generalizing poorly to unseen ones.

Approach

Structured CoT (visual analysis, candidate sub-categories, comparison, prediction) distilled from a larger MLLM, followed by a DAPO-based policy optimization that adds triplet (anchor, positive, hard negative) augmentation: mixing rollouts from same-class images and maximizing a KL divergence between anchor- and negative-conditioned policies.

Key results

• Fine-R1-7B reaches 88.71% average closed-world accuracy, versus 80.20% for Qwen2.5-VL-7B and 84.44% for SigLIP-L.
• Fine-R1-7B achieves 74.80% average open-world relative semantic similarity, about 23.75 points above Qwen2.5-VL-7B.
• Fine-R1-3B improves unseen-category closed-world accuracy to 81.41%, and TAPO gives +10.05% over No-Thinking RL on unseen categories.

Surprising findings

• Direct zero-shot CoT prompting without training barely changes accuracy (74.79% vs 74.68% baseline), suggesting reasoning alone does not help FGVR.
• Linear probing and taxonomy-similarity analyses show Fine-R1 does not improve visual features or subcategory knowledge; gains come from better deployment of existing knowledge.

Limitations

• Open-world evaluation relies on text inclusion and SigLIP-based similarity, which can be imperfect for granularity mismatches.
• CoT SFT data is small (404 samples) and relies on manual verification and correction.

Experimental setup

Datasets: CUB-200 Stanford Cars Stanford Dogs Flowers-102 Oxford-IIIT Pet FGVC-Aircraft

Benchmarks: CUB-200 Stanford Cars Stanford Dogs Flowers-102 Oxford-IIIT Pet FGVC-Aircraft ImageWikiQA MME MMBench SEED-Bench

Metrics: accuracy text inclusion relative semantic similarity

Baselines: CLIP-L EVA-G SigLIP-L SigLIP2-L Qwen2.5-VL-7B DeepPerception-7B CLS-RL No-Thinking RL

Builds on: DAPO GRPO CLS-RL PAPO

Closest papers by meaning

• Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.93)
• Democratizing Fine-grained Visual Recognition with Large Language Models (ICLR 2024 · similarity 0.92)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
• Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.89)
• GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs (ICLR 2026 · similarity 0.89)
• DiVE-k: DIFFERENTIAL VISUAL REASONING FOR FINE-GRAINED IMAGE RECOGNITION (ICLR 2026 · similarity 0.89)