Research map / Vision-language and multimodal / Multimodal LLMs

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoVAND, a CoT and VQA-verified negation dataset of region-grounded captions, and NegToMe, a negation-boosted text token merging module combined with LoRA on deep cross-attention, to fix affirmative bias in described object detection.

Problem

Vision-language detectors suffer from affirmative bias: they fail to distinguish described objects with negation (e.g., "person without hat") from their affirmative counterparts, due to scarce negation data and fragmented tokenization that weakens negation cues.

Approach

Combine a region-grounded negation dataset built via three-step GPT-4o chain-of-thought captioning and region-level VQA alignment with NegToMe, which merges negation cues with their attribute phrases into single tokens via a parser and up-weights the negation cue in the merged embedding, plus LoRA on deep text cross-attention layers.

Key results

• On D^3, Grounding DINO-B with the method improves full mAP from 20.7 to 27.3 (+6.6) and absence-subset mAP by +7.2.
• On OVDEval Negation, Grounding DINO-B NMS-AP rises from 36.8 to 47.6 (+10.8).
• Qwen-2.5-VL-3B improves from 18.6 to 22.2 mAP on D^3 and from 34.6 to 41.9 AP on OVDEval Negation.

Surprising findings

• Qwen-2.5-VL-7B underperforms the much smaller Grounding DINO-B baseline on OVDEval-Negation (35.9 NMS-AP vs 36.8).

Limitations

• CoVAND annotation error rate estimated at 2.3% on a 1,000-sample audit; negation cues are dominated by regularized forms (83.41%).
• Dense scenes with many same-type instances are ambiguous, so images with more than five instances were excluded.
• Model sometimes misses target instances on complex negations (e.g., "pizza that is not complete").

Experimental setup

Datasets: Flickr30k Entities CoVAND Flickr30k

Benchmarks: D^3 OVDEval (Negation subset) NegBench MCQ (COCO) FG-CXR

Metrics: mAP AP NMS-AP FPR accuracy

Baselines: Grounding DINO-B APE-Ti Qwen-2.5-VL-3B SPHINX-7B OFA-DOD FIBER-B GLIP-T OWL-ViT-L

Builds on: Grounding DINO LoRA Token Merging (ToMe) APE

Closest papers by meaning

• Vision-Language Models Do Not Understand Negation (CVPR 2025 · similarity 0.91)
• Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness (CVPR 2026 · similarity 0.87)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.87)
• Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
• The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.87)
• VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning? (ICLR 2026 · similarity 0.86)