Research map / Architectures and efficiency / RNNs and normalization
Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a spectral-clustering-inspired gradient-based optimization over key-query and value-value affinity matrices from all attention layers of pre-trained vision models, extracting per-image and dataset-level segmentation and revealing 'what' (value) and 'where' (key-query) pathways in Stable Diffusion.
Problem
Extracting spatial layout and semantic segmentation from pre-trained vision models without knowing which layer holds the relevant information, and understanding how attention layers encode semantic versus spatial information.
Approach
Jointly optimize a learnable feature map against a set of per-layer (and per-timestep) affinity matrices built from attention queries, keys, and values, approximating constrained spectral clustering via a Rayleigh-quotient objective solved with gradient descent, so it scales to dataset-level intra- and inter-image affinity graphs.
Key results
- Ours + K-Means on Stable Diffusion 1.4 all attentions reaches 0.82 mIoU oracle segmentation on PASCAL VOC, above the CLIP and DINO baselines.
- Unsupervised semantic segmentation with the value-value graph reaches 25.4 mIoU on COCO-Stuff-27, close to STEGO's 26.8 (ViT-B/8).
- Key-query graph gives coordinate regression MSE of 19.5 versus 43.1 for the value-value graph, supporting a 'where' pathway.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Analyze all attention layers instead of only final features | Ours + K-Means on CLIP ViT-B/16 · PASCAL VOC val | mIoU: 0.57 -> 0.78 | Helped |
| Analyze all attention layers instead of only final features | Ours + K-Means on MAE ViT-B/16 · PASCAL VOC val | mIoU: 0.48 -> 0.74 | Helped |
| Restrict analysis to decoder layers 11-16 (ablation) | Stable Diffusion region extraction · PASCAL VOC | mIoU: 0.75 -> 0.80 | Helped |
| Restrict analysis to encoder layers 1-4 (ablation) | Stable Diffusion region extraction · PASCAL VOC | mIoU: 0.75 -> 0.66 | Hurt |
| Set maximum noise level to 999 instead of 500 (ablation) | Stable Diffusion region extraction · PASCAL VOC | mIoU: 0.75 -> 0.69 | Hurt |
| Add CRF post-processing to the value-value graph output (ablation) | Ours (V-V graph) · COCO-Stuff-27 | mIOU: 25.4 -> 27.1 | Helped |
| Add CRF post-processing to the value-value graph output (ablation) | Ours (V-V graph) · Cityscapes | mIOU: 16.2 -> 16.9 | Helped |
| Use query-key graph instead of value-value graph (ablation) | Unsupervised semantic segmentation · COCO-Stuff-27 | mIOU: 25.4 -> 12.4 | Hurt |
| Use query-key graph instead of value-value graph (ablation) | Unsupervised semantic segmentation · Cityscapes | mIOU: 16.2 -> 10.91 | Hurt |
| Use query-key graph instead of value-value graph (ablation) | Stable Diffusion features · Spatial semantic segmentation | mIOU: 5.1 -> 10.1 | Helped |
Surprising findings
- Key-query and value-value affinity graphs from the same generative model split cleanly into spatial and semantic eigenvector sets, despite the projections not being constrained to do so.
- The value-value graph reaches near-STEGO performance on COCO-Stuff-27 using a generative diffusion backbone and only mini-batch neighbors.
Limitations
- Unsupervised segmentation on Cityscapes lags STEGO (16.2 vs 18.2 mIoU without CRF), and Hungarian matching makes the Q-K graph and STEGO comparison less favorable in some settings.
- Per-image extraction takes about 67 seconds per image on Stable Diffusion, and dataset-level optimization needs substantial compute.
- Many hyperparameters (layer subsets, noise level, K) are chosen by ablation on the evaluation data.
Experimental setup
Datasets: PASCAL VOC COCO COCO-Stuff Cityscapes
Benchmarks: PASCAL VOC COCO-Stuff-27 Cityscapes
Metrics: mIoU MSE
Baselines: STEGO PiCIE DSM MaskCut TokenCut Normalized Cut HP ACSeg DINO K-Means
Builds on: spectral clustering normalized cuts STEGO TokenCut MaskCut
Closest papers by meaning
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.87)
- Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models (NeurIPS 2026 · similarity 0.86)
- Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation (AAAI 2025 · similarity 0.85)
- Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.85)
- Latent Functional Maps: a spectral framework for representation alignment (NeurIPS 2024 · similarity 0.85)
- Dissecting Query-Key Interaction in Vision Transformers (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.