Research map / Vision-language and multimodal / Multimodal LLMs

SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SynDORBench, a Blender-rendered benchmark of about 54k QA pairs that tests LVLM human-presence classification and action recognition under DORI-calibrated distances, lighting, and night-vision conditions, and compares 16 open-source LVLMs, GPT-4o-mini, and YOLO11x.

Problem

LVLM evaluations assume ideal, high-visibility inputs and do not measure how physical imaging constraints (distance, illumination, viewpoint, pixel density) degrade semantic recoverability relative to human visibility standards such as DORI.

Approach

Anchors synthetic rendering to the BS EN 62676-4 DORI pixel-density thresholds via pinhole camera geometry, so distances map to physically interpretable px/m regimes, and adds a discernibility labeling pipeline (RF+XGBoost ensemble on mask-conditioned features) for low-light supervision.

Key results

• At 5.4 m, gemma-3n-E2B-it reaches 87.10% accuracy and 75.78 F1 on HPC, while YOLO11x reaches 73.92% accuracy and 50.17 F1; YOLO11x falls to 50.29% accuracy at 23 m.
• Under night (no enhancement) conditions, no model reaches an F1 of 0.8, and the paper describes all LVLMs as effectively non-operational in that setting.
• On action recognition, the MVLM group mean (0.83 at 5.4 m) is similar to GPT-4o-mini (0.83), and all LVLMs score below the non-expert human (0.94 at 5.4 m).

Surprising findings

• Compact open-source models such as gemma-3-4b-it and gemma-3n-E2B-it match or outperform larger MVLMs and GPT-4o-mini on HPC, so robustness does not scale monotonically with parameter count.
• YOLO11x falls below the 80% operational threshold already at 5.4 m, where visibility is relatively favorable.

Limitations

• Single-human, frame-level scenarios only, which limits realism for multi-person or video settings.
• Synthetic renderings may not transfer to real camera imagery.
• Night-vision subset is produced by a vendor-neutral ISP approximation rather than real camera pipelines.

Experimental setup

Datasets: BABEL AMASS SMPLitex BlenderKit assets

Benchmarks: SynDORBench-54k

Metrics: accuracy macro-F1 E5Score z-score

Baselines: YOLO11x GPT-4o-mini

Builds on: DORI (BS EN 62676-4:2015) BABEL SMPL-X

Closest papers by meaning

• MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs (ICML 2026 · similarity 0.91)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.89)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.89)
• LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.89)