Research map / Vision-language and multimodal / Multimodal LLMs

DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DRBench (14,573 dense-scene VQA questions with structured reasoning annotations) and DRScaffold, a four-stage staged-gradient-masking SFT scheme that lets lightweight VLMs (2-5.6B) ground entities, relations, and reasoning before answering.

Problem

Lightweight VLMs perform well on standard benchmarks but fail at dense-scene reasoning requiring joint grounding of multiple objects, attributes, and relations; existing supervision (answer-only or flat CoT) gives no grounding between reasoning steps and visual entities.

Approach

Decompose the supervision target into four causally ordered fields (key entities, scene graph, reasoning chain, answer) and unlock them progressively via staged gradient masking during LoRA SFT, with no architectural change, plus a benchmark with structured intermediate annotations and a false-premise category.

Key results

• Qwen2.5-VL-3B with DRScaffold reaches 62.6 on DRBench vs 35.0 base and 44.7 for Qwen2.5-VL-32B.
• InternVL2.5-2B improves DRBench from 32.5 to 66.6 with DRScaffold while MMStar drops 1.5 points.
• Versus CoT Supervision (57.8) and LLaVA-CoT (57.1) on DRBench, DRScaffold achieves 62.6 while keeping general benchmark averages at 52.9 vs 50.3 for baselines.

Surprising findings

• Answer-only and CoT supervision degrade general benchmarks (e.g., MMMU 47.1 to 44.3) even while improving DRBench.
• After Stage 1 alone the model produces correct chain-of-thought but still wrong direct answers, a disconnect the paper attributes to missing relational structure.

Limitations

• DRBench covers only indoor synthetic (Hypersim) and urban driving (Cityscapes) scenes.
• Gains on Phi4-multimodal are modest, suggesting diminishing returns for stronger base models.
• Staged training multiplies total training time with the number of stages.

Experimental setup

Datasets: DRBench Hypersim Cityscapes LLaVA-Instruct-150K

Benchmarks: MMMU MMStar RealWorldQA HallusionBench BLINK DRBench

Metrics: accuracy figure-level accuracy (fAcc) scene graph F1 reasoning validity

Baselines: Answer Supervision CoT Supervision LLaVA-CoT Qwen2.5-VL-32B

Builds on: LLaVA-CoT LoRA LLaVA-Instruct-150K

Closest papers by meaning

• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.89)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.89)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.88)
• VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction (CVPR 2026 · similarity 0.88)
• Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding (ICML 2026 · similarity 0.88)