Research map / Vision-language and multimodal / Multimodal LLMs
FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FineRS, a two-stage RL-trained MLLM (coarse region selection then local box/point refinement with frozen SAM2) and the FineRS-4k UAV 4K benchmark for small-object reasoning and segmentation.
Problem
MLLMs struggle to localize and segment extra-small objects in high-resolution (4K) cluttered images due to restricted input resolution and reliance on large supervised datasets.
Approach
A two-stage Qwen2.5-VL-7B pipeline: Global Semantic Exploration predicts a coarse region on the full image, Localized Perceptual Refinement predicts box and points on a crop, and frozen SAM2 produces the mask; both trained with GRPO using task-specific rewards, plus a locate-informed retrospective reward that uses LPR outputs to supervise GSE.
Key results
- On the FineRS-4k test set, FineRS-7B reaches 55.1 gIoU / 46.5 cIoU, versus 46.6 gIoU for retrained Seg-Zero-7B.
- On V*, FineRS achieves 77.5% overall accuracy, above SEAL (75.4%) and reported DC2 (57.3%) on that benchmark.
- On HR-Bench 4K, FineRS reaches 63.8% average accuracy versus 50.0 for DC2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the retrospective reward to GSE training (ablation) | FineRS · FineRS-4k test | gIoU: 54.0 -> 55.1 | Helped |
| add random input region augmentation during LPR training (ablation) | FineRS · FineRS-4k test | gIoU: 53.9 -> 55.1 | Helped |
| add QA accuracy reward (ablation) | FineRS · FineRS-4k test | gIoU: 52.8 -> 55.1 | Helped |
| add box size reward (ablation) | FineRS · FineRS-4k test | gIoU: 51.0 -> 55.1 | Helped |
| add box-in-region reward (ablation) | FineRS · FineRS-4k test | gIoU: 50.1 -> 55.1 | Helped |
| use a two-stage coarse-to-fine design instead of One Stage without QA reward (ablation) | FineRS · FineRS-4k test | gIoU: 46.6 -> 55.1 | Helped |
| apply QA accuracy reward in GSE only rather than in both GSE and LPR (ablation) | FineRS · FineRS-4k test | gIoU: 55.1 -> 53.7 | Hurt |
| apply QA accuracy reward in LPR only rather than in both GSE and LPR (ablation) | FineRS · FineRS-4k test | gIoU: 55.1 -> 54.4 | Hurt |
| use group size 6 instead of 8 (ablation) | FineRS · FineRS-4k test | gIoU: 55.1 -> 54.3 | Hurt |
| use group size 4 instead of 8 (ablation) | FineRS · FineRS-4k test | gIoU: 55.1 -> 52.4 | Hurt |
| use KL weight 5e-2 instead of 5e-3 (ablation) | FineRS · FineRS-4k test | gIoU: 55.1 -> 54.4 | Hurt |
| use Seed 48 instead of Seed 42 (ablation) | FineRS · FineRS-4k test | gIoU: 55.1 -> 53.9 | Hurt |
Surprising findings
- Seg-Zero (training-free) outperforms FineRS on small objects in the validation set (67.2 vs 64.8 gIoU on S), as noted by the authors.
Limitations
- LPR localization depends on the coarse region from GSE; if the target falls outside the region, the answer and box are wrong.
- Two stages are not jointly optimized end-to-end due to memory constraints.
- Higher inference latency than training-free baselines.
Experimental setup
Datasets: FineRS-4k
Benchmarks: FineRS-4k V* HR-Bench 4K HR-Bench 8K ReasonSeg
Metrics: gIoU cIoU MVQA accuracy OVQA accuracy (fuzzy match) accuracy inference time
Baselines: LISA LISA++ PixelLM SEAL DC2 MLLMs-Know Seg-Zero VILA-HD
Builds on: Seg-Zero Visual-RFT LISA GRPO SAM2
Closest papers by meaning
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- LENS: Learning to Segment Anything with Unified Reinforced Reasoning (AAAI 2026 · similarity 0.89)
- DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.89)
- SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.