Research map / Vision-language and multimodal / Multimodal LLMs

ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ORIC, a framework using LLM-guided and CLIP-guided sampling to build object-recognition questions under contextual incongruity, releases ORIC-Bench, and shows Visual-RFT of Qwen3-VL-8B on 600 ORIC-style samples improves ORIC, AMBER, and HallusionBench.

Problem

LVLMs miss present objects and hallucinate absent ones when object-context pairs are incongruous, a high-uncertainty regime not covered by existing benchmarks like POPE, AMBER, and HallusionBench.

Approach

Construct incongruous object-context binary questions from MSCOCO: GPT-5 selects present small objects that are implausible given the other scene objects (LLM-guided), and CLIP retrieves a similar image and selects plausible but absent objects (CLIP-guided); then apply GRPO-based Visual-RFT with verifiable accuracy and format rewards.

Key results

• Visual-RFT of Qwen3-VL-8B-Instruct raises ORIC-Bench macro F1 from 79.55 to 82.79 and human-labeled macro F1 from 78.63 to 83.62.
• Qwen3-VL-8B-Instruct achieves the top ORIC-Bench macro F1 of 79.55 among 18 LVLMs, slightly above GPT-5 (78.61); most models fall between 60 and 77 F1.
• On AMBER, the Visual-RFT model improves accuracy from 87.48 to 90.49, while HallusionBench stays roughly stable (69.37 to 69.81).

Surprising findings

• Qwen3-VL-8B-Instruct and GPT-5 score near 100 macro F1 on POPE subset but drop to around 60 on incongruous-context versions of the same images.
• Open-source Qwen3-VL-8B-Instruct slightly outperforms closed-source GPT-5 on ORIC-Bench.

Limitations

• Limited to a single dataset (MSCOCO) and a binary yes/no existence formulation.
• Relies on CLIPScore and GPT-5 as proxies for context plausibility, with CLIP co-occurrence bias acknowledged.
• Human-labeled subset is small (200 questions) and the human labels are authored by the same team.

Experimental setup

Datasets: MSCOCO 2014 train MSCOCO 2014 val POPE AMBER HallusionBench

Benchmarks: ORIC-Bench AMBER HallusionBench POPE

Metrics: macro precision macro recall macro F1 yes proportion per-class F1 recall by object size

Baselines: GPT-5-2025-08-07 InternVL3-9B Janus-Pro-7B Grounding DINO 1.5 Pro OWLv2 Emu3-Chat Qwen3-VL-8B-Instruct 0-shot CoT

Builds on: Visual-RFT GRPO POPE CLIP DeepSeek-R1

Closest papers by meaning

• MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.89)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.88)
• Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
• Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models (ICLR 2025 · similarity 0.87)