Research map / Vision-language and multimodal / Human and embodied modeling
Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RepDecompose and CompAlign to decompose ViT final representations into component contributions, map them to CLIP space via orthogonal-regularized linear maps, and interpret/rank components by feature via a scoring function.
Problem
Understanding the roles of attention heads and MLPs in arbitrary vision transformers (beyond CLIP-ViT), which lack text encoders and have diverse attention and resampling operations that complicate decomposition.
Approach
Automatically decompose the representation by traversing the computation graph to obtain component contributions for any ViT, then align each contribution to CLIP space with per-component linear maps regularized toward orthogonality, enabling text interpretation and a continuous component-feature scoring function.
Key results
- CompAlign's TextSpan descriptions match 0.185 of ImageNet pre-trained descriptions on DeiT-B/16 vs 0.08 for a single shared map.
- Zero-shot spurious correlation mitigation by ablating 10 location components raises DeiT worst-group accuracy from 0.733 to 0.815 and CLIP from 0.507 to 0.744 on Waterbirds.
- Zero-shot segmentation via decomposition gives DeiT mIoU 0.5291 vs 0.4785 for Chefer et al. and 0.4625 for GradCAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use separate component maps with the orthogonality regularizer rather than one unconstrained map (ablation) | CompAlign on DeiT-B/16 · ImageNet-1k | match rate: 0.08 -> 0.185 | Helped |
| Add the orthogonality regularizer to separate component maps (ablation) | CompAlign on DeiT-B/16 · ImageNet-1k | match rate: 0.155 -> 0.185 | Helped |
| Add the orthogonality regularizer to separate component maps (ablation) | CompAlign on DeiT-B/16 · ImageNet-1k | cosine distance: 0.18 -> 0.17 | Helped |
| Use separate component maps instead of one unconstrained map (ablation) | CompAlign on DeiT-B/16 · ImageNet-1k | cosine distance: 0.23 -> 0.17 | Helped |
| Apply component decomposition for zero-shot segmentation | DeiT · ImageNet-1k | pixAcc: 0.6533 -> 0.7719 | Helped |
| Apply component decomposition for zero-shot segmentation | DINO · ImageNet-1k | mIoU: 0.4309 -> 0.4863 | Helped |
| Apply component decomposition for zero-shot segmentation | MaxVit · ImageNet-1k | mAP: 0.4243 -> 0.7237 | Helped |
| Ablate the top 10 location-associated, non-bird components (ablation) | DeiT · Waterbirds | worst group accuracy: 0.733 -> 0.815 | Helped |
| Ablate the top 10 location-associated, non-bird components (ablation) | CLIP · Waterbirds | worst group accuracy: 0.507 → 0.744 | Helped |
| Ablate the top 10 location-associated, non-bird components (ablation) | DINO · Waterbirds | worst group accuracy: 0.800 → 0.911 | Helped |
| Ablate the top 10 location-associated, non-bird components (ablation) | DINOv2 · Waterbirds | worst group accuracy: 0.967 → 0.978 | Helped |
| Ablate the top 10 location-associated, non-bird components (ablation) | SWIN · Waterbirds | worst group accuracy: 0.834 → 0.871 | Helped |
Surprising findings
- Individual components in larger models MaxViT and Swin do not respond strongly to any single image feature but encode features in combination.
- For models not trained on ImageNet (CLIP, DINOv2), removing the last few layers quickly drops accuracy to zero, while ImageNet-trained models degrade gradually, indicating redundancy.
Limitations
- Only direct contributions from the last few layers are analyzed; higher-order indirect contributions are not studied.
- Decomposition is restricted to attention heads and tokens; convolutional blocks are not decomposed.
- Unclear whether specific directions or subspaces within component contributions correspond to properties.
Experimental setup
Datasets: ImageNet-1k Waterbirds LVD-142M (pretraining of DINOv2, not used directly)
Benchmarks: ImageNet-1k Waterbirds
Metrics: top-1 accuracy Spearman rank correlation cosine distance match rate pixAcc mIoU mAP worst group accuracy average group accuracy
Baselines: Chefer et al. zero-shot segmentation GradCAM single shared linear map CompAlign with lambda=0
Builds on: CLIP-ViT component decomposition (Gandelsman et al.) TextSpan model stitching
Closest papers by meaning
- Interpreting CLIP's Image Representation via Text-Based Decomposition (ICLR 2024 · similarity 0.90)
- CLIP in Mirror: Disentangling text from visual images through reflection (NeurIPS 2024 · similarity 0.88)
- Gradient-based Visual Explanation for Transformer-based CLIP (ICML 2024 · similarity 0.87)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.86)
- Rejuvenating image-GPT as Strong Visual Representation Learners (ICML 2024 · similarity 0.86)
- Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE) (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.