Research map / Vision-language and multimodal / Multimodal LLMs

COCOTree: A Dataset and Benchmark for Open Tree-Structured Visual Decomposition

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces open tree decomposition and COCOTree, a 21K-image COCO-based dataset of open-vocabulary hierarchical mask trees built by an automated LVLM + SAM 3 recursive pipeline, plus the OTQ metric for evaluating such trees.

Problem

Existing hierarchical segmentation datasets and metrics are limited to shallow, fixed-depth, closed-vocabulary part hierarchies, and manual annotation of unconstrained hierarchical structures is prohibitively costly.

Approach

A recursive pipeline in which an LVLM proposes open-vocabulary child labels for each (cropped) parent region and SAM 3 grounds them into instance masks, producing unconstrained-depth open trees; paired with a PQ-style metric (OTQ) that jointly scores mask IoU, label similarity, and parent-structure consistency.

Key results

• COCOTree contains over 21K images, 1.8M instance nodes, 85.7 masks per image, average depth 3.448, and over 3.5K unique open-vocabulary labels.
• Human validation (20 reviewers, 50 images each) gives >=91% Good ratings for label-mask match, instance separation, child validity, and leaf appropriateness; tree-level consistency has 47.5% Good and 44.3% Minor.
• Recursive SAM 3 tree achieves OTQ 0.587 and HPQ 0.349 versus 0.098-0.119 OTQ for flat COCO-family projections on the benchmark.

Surprising findings

• Flat projections of existing COCO-based datasets score very low on tree quality (TQ about 0.13-0.15) despite nonzero mask and label quality, showing they lack image-specific tree structure.
• Mask recall for extra-small masks is low (AR_XS about 0.13 for COCO-17), and LVIS median IoU is only 0.73 versus 0.90 for COCO-ReM.

Limitations

• Automated annotations contain geometric and structural noise, with missed small children, noisy masks, and duplicates concentrated in deep regions.
• Open-vocabulary label evaluation relies on similarity backends (BERT, OEWN, Qwen) that may introduce synonym and parent-dependent errors.
• Multiple valid decompositions exist per image, so a single reference tree is only one consensus solution.

Experimental setup

Datasets: COCO

Benchmarks: COCOTree

Metrics: OTQ HPQ TQ meanNQ MQ LQ IoU AR

Baselines: COCO-17 COCONut COCO-ReM LVIS COCO-Stuff PACO CPP PartImageNet SPIN ADE20K

Builds on: SAM 3 Panoptic Quality COCO

Closest papers by meaning

• Image2Struct: Benchmarking Structure Extraction for Vision-Language Models (NeurIPS 2024 · similarity 0.86)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.86)
• MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.86)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.86)
• Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.86)
• Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thought (ICML 2026 · similarity 0.86)