Research map / Vision-language and multimodal / Multimodal LLMs

SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SegCompass, an end-to-end reasoning segmentation model that routes CoT and visual tokens through a pretrained sparse autoencoder, a query codebook and slot mapper to a multi-slot heatmap, trained jointly with GRPO and segmentation losses.

Problem

Reasoning segmentation pipelines either use opaque latent query alignment or textual localization readouts that are not truly interpretable or tightly connected to perception.

Approach

Uses a sparse autoencoder over both CoT and visual token hidden states to form a shared sparse concept space; a learned query codebook selects concepts, and a slot mapper grounds them into multi-slot heatmaps that guide the mask decoder, with joint GRPO and segmentation supervision.

Key results

• SegCompass-13B reaches 84.0 cIoU on RefCOCOg val and 81.3 on test, with 66.5 gIoU on ReasonSeg test in zero-shot setting (the test-split figure is cIoU in the table, per its column header; text reports ReasonSeg gIoU/cIoU).
• Ablation on RefCOCOg/gRefCOCO/ReasonSeg: RL-only 65.9/63.0/40.1, segmentation-only 77.9/74.0/59.3, combined 81.3/77.3/66.5.
• SegCompass-7B (Qwen2.5-VL) reaches 85.3 cIoU on RefCOCO val and 83.6 on testB, competitive with HiMTok and X-SAM.

Surprising findings

• Reinforcement learning alone performs far worse than segmentation supervision alone (e.g., ReasonSeg 40.1 vs 59.3), though combining them gives the best results.

Limitations

• Interpretability is evaluated mainly via activation statistics and instance coverage; no human study of concept quality is reported.
• Gains over strongest baselines (e.g., X-SAM, HiMTok) are mixed; SegCompass does not uniformly surpass them on all splits.

Experimental setup

Datasets: OBELICS RefCOCO RefCOCO+ RefCOCOg gRefCOCO

Benchmarks: RefCOCO RefCOCO+ RefCOCOg gRefCOCO ReasonSeg

Metrics: cIoU gIoU

Baselines: LISA PerceptionGPT PixelLM GLaMM UniRES HiMTok X-SAM Seg-Zero VisionReasoner SAM-R1 Text4Seg Seg-R1

Builds on: LISA GRPO SAM SAE-V Seg-Zero

Closest papers by meaning

• Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search (ICML 2026 · similarity 0.89)
• DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models (ICML 2026 · similarity 0.89)
• CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation (ICLR 2026 · similarity 0.88)
• Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design (CVPR 2026 · similarity 0.88)
• Fast Reasoning Segmentation for Images and Videos (CVPR 2026 · similarity 0.88)
• Discriminative Perception via Anchored Description for Reasoning Segmentation (CVPR 2026 · similarity 0.87)