Research map / Detection and segmentation / Open-set and few-shot detection

SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis

NeurIPS 2026 · Detection and segmentation › Open-set and few-shot detection · approach: Open-vocabulary 3D and world perception · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SANEval, an open-vocabulary benchmark for compositional text-to-image evaluation (attribute binding, spatial relations, numeracy) that pairs LLM prompt parsing and LLM-expanded synonyms with YOLO-E detection and VLM-as-judge scoring, with diagnostic feedback.

Problem

Existing T2I compositional benchmarks rely on fixed-vocabulary detectors (e.g., 80 MS-COCO classes), produce single opaque scores, and are rarely validated against human judgment, so they cannot evaluate arbitrary objects or diagnose specific failures.

Approach

Combine an LLM for structured prompt parsing and synonym expansion with an open-vocabulary YOLO-E detector, then score spatial, numeracy and attribute adherence with geometric rules, counts, and VLM-based attribute checks, emitting per-item conformity feedback.

Key results

• Imagen 4.0 Ultra achieves the highest SANEval-Simple average (0.4992 across attribute, spatial, numeracy), with Seedream 3.0 best on spatial and color binding.
• Replacing Gemini Flash 2.5 with Llama-4 as the LLM backbone changes average scores by about -0.046, a small shift the authors take as evidence of judge robustness.
• Removing LLM synonym expansion drops spatial scores from 0.4032 to 0.0980 (75.69% relative drop) in the ablation.

Surprising findings

• Human ratings rank GPT Image 1 highest on numeracy and shape binding, where SANEval ranks Imagen 4.0 Ultra highest.
• Several categories show Spearman correlations with CompBench++ with p-values above 0.05, so the two benchmarks disagree on those categories.

Limitations

• Relies on proprietary Gemini 2.5 Flash as VLM and LLM judge, with limited open-model comparison.
• Human evaluation covers only 100 responses per benchmark, and category-level human rankings diverge from automated scores.
• Spatial relations limited to eight geometric relations judged from bounding-box centroids.

Experimental setup

Datasets: SANEval-Simple SANEval-Hard

Benchmarks: SANEval-Simple SANEval-Hard

Metrics: average score Spearman rank correlation human agreement

Baselines: CompBench++ GenEval Llama-4 (LLM backbone swap)

Builds on: GenEval CompBench++ YOLO-E Davidsonian Scene Graph ConceptMix

Closest papers by meaning

• Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2) (NeurIPS 2024 · similarity 0.91)
• ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty (NeurIPS 2024 · similarity 0.89)
• FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.89)
• Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.88)
• DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.88)
• Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.87)