Research map / Vision-language and multimodal / Multimodal LLMs
PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PARTONOMY, a part-level grounding benchmark for LMMs (Explanatory Part Segmentation) and PLUM, a segmenting LMM using BIO span tagging and a mask feedback loop instead of [SEG] tokens.
Problem
Large multimodal models struggle to identify, ground, and reason about object parts at pixel level; existing segmenting LMMs use special [SEG] tokens and discard past mask predictions, which hurts general reasoning and part grounding.
Approach
Replace special segmentation tokens with a bidirectional BIO span extractor over the LLM's text outputs, add a KL constraint to keep span embeddings near the frozen backbone, and condition later mask predictions on earlier masks via a FiLM-modulated mask encoder feedback loop; pair this with a benchmark built from part datasets plus a manually annotated core split with multiple-choice explanatory
Key results
- PLUM-13B zero-shot reaches 14.5 micro-gIoU on Identification vs. 5.9 for LISA-13B on PARTONOMY-Core.
- PLUM-13B achieves 30.11% on TextVQA, a 31.8% relative gain over the LLaVA-13B backbone (22.84), while LISA-13B drops to 1.58.
- PLUM-13B (ft) reaches 57.3 gIoU on ReasonSeg, above LISA-13B (ft) at 56.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add mask feedback loop (FiLM-conditioned prior-mask pooling) to span-tagging model (ablation) | PLUM-13B · Partonomy-Core | micro-gIoU: -9.6% when removed | Helped |
| Replace [SEG] tokens with BIO span tagging (span extractor) and KL-regularized queries | PLUM-13B vs LISA-13B · Partonomy-Core zero-shot | micro-gIoU: 14.5 vs 5.9 | Helped |
| Fine-tune on Partonomy training splits | PLUM-13B · Partonomy-Core Identification | macro-gIoU: 27.4 to 41.6 | Helped |
| Use Focal-Tversky loss instead of Dice (ablation) | PLUM-13B · PartImageNet (Partonomy val) | micro-gIoU: 66.47 to 67.90 | Helped |
| Remove bidirectional attention from BIO span extractor (ablation) | PLUM span tagger · Partonomy-PartImageNet and RefCOCO | I-tag accuracy: 87.32 to 15.86 (Partonomy) | Hurt |
| Replace [SEG] tokens with span tagging (no special tokens) | LLaVA-13B-based segmenting LMM · TextVQA / GQA / POPE | accuracy: TextVQA +31.8%, GQA -23.6%, POPE +8.9% vs LLaVA-13B | Mixed |
| Use [SEG] token-based segmentation (LISA) with LLaVA-13B | LISA-13B · TextVQA | accuracy: 22.84 to 1.58 | Hurt |
| Increase KL weight lambda_KL from 0 to 1.0 (ablation) | PLUM · PartImageNet segmentation vs TextVQA | micro-gIoU and TextVQA accuracy: PartImageNet micro-gIoU falls ~20%; TextVQA improves ~75% | Mixed |
| Ablate span extractor and feedback loop jointly (ablation) | PLUM · Partonomy-Core | macro-gIoU: span extractor alone beats LISA by 8.4% macro-gIoU | Helped |
| Add ReasonSeg finetuning to PLUM | PLUM-13B · ReasonSeg | gIoU: 57.3 vs LISA-13B (ft) 56.2 | Helped |
Surprising findings
- Segmenting LMMs such as LISA and PixelLM lose most of their VQA and hallucination ability after segmentation finetuning, with LISA scoring near zero on TextVQA and GQA.
- GPT-4o reaches 81.7% identification accuracy, far above all segmenting LMMs, though it is given all answer choices at once.
Limitations
- Core split omits some rare or domain-specific parts and lacks object diversity of larger datasets.
- Struggles with small or ambiguous parts.
- May not scale efficiently to high-resolution imagery.
Experimental setup
Datasets: PASCAL-Part PartImageNet PACO COCO-Stuff ADE20k RefCOCO llava_instruct_150k ReasonSeg
Benchmarks: PARTONOMY-Core PARTONOMY-PACO_LVIS PARTONOMY-PartImageNet PARTONOMY-PascalParts ReasonSeg TextVQA GQA POPE
Metrics: gIoU micro-gIoU macro-gIoU accuracy precision recall
Baselines: LISA-13B GLaMM PixelLM-13B SegLLM X-Decoder SEEM Grounded SAM 2 GPT-4o
Builds on: LISA LLaVA SAM GLaMM PixelLM
Closest papers by meaning
- CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models (CVPR 2025 · similarity 0.89)
- PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understanding (NeurIPS 2025 · similarity 0.89)
- Fine-Grained Image-Text Correspondence with Cost Aggregation for Open-Vocabulary Part Segmentation (CVPR 2025 · similarity 0.88)
- Compositional Chain-of-Thought Prompting for Large Multimodal Models (CVPR 2024 · similarity 0.88)
- LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation (NeurIPS 2025 · similarity 0.88)
- Understanding Multi-Granularity for Open-Vocabulary Part Segmentation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.