Research map / Vision-language and multimodal / Cross-modal learning
Large Multi-modal Models Can Interpret Features in Large Multi-modal Models
Kaichen Zhang, Yifei Shen, Bo Li, Ziwei Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAE feature explanations from the V-Interp pipeline instead of random images | SAE features · SAE explanations (total) | IoU: 0.005 ± 2 × 10^-4 -> 0.20 | Improved |
| use SAE feature explanations from the V-Interp pipeline instead of random images | SAE features · SAE explanations (total) | CLIP score: 18.2 ± 1 × 10^-2 -> 23.6 | Improved |
| hook the SAE at a deeper LLaVA layer (8th to 32nd) (ablation) | SAE on LLaVA-NeXT-LLaMA3-8B · Cross-layer SAE | IoU: 0.30 -> 0.40 | Improved |
| hook the SAE at a deeper LLaVA layer (8th to 32nd) (ablation) | SAE on LLaVA-NeXT-LLaMA3-8B · Cross-layer SAE | CLIP score: 22.82 -> 26.55 | Improved |
| hook the SAE at the 25th instead of the 8th layer (ablation) | SAE on LLaVA-NeXT-LLaMA3-8B · Cross-layer SAE | IoU: 0.30 -> 0.31 | No clear effect |
| apply the pipeline to InstructBLIP-7B instead of LLaVA (ablation) | V-Interp · InstructBLIP-7B SAE | CLIP score: 17.71 -> 28.01 | Improved |
| apply the pipeline to Qwen2.5-VL (ablation) | V-Interp · Qwen2.5-VL SAE | IoU: 0.06 -> 26.67 | Improved |
| steer with identified features instead of random features | LLaVA-NeXT · Steering effect evaluation | GPT-4o score: 2.02 -> 6.36 | Improved |
| clamp a single OCR-related feature to a high value | LLaVA-NeXT · HallusionBench (100 examples) | fraction of cases improved: 0.09 improved, 0.02 worse, 0.89 same | Improved |
| scale SAE from 2^17 to 2^18 features (ablation) | SAE on LLaVA-NeXT · SAE training loss | loss | No clear effect |
Closest papers by meaning
- L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.88)
- A Concept-Based Explainability Framework for Large Multimodal Models (NeurIPS 2024 · similarity 0.88)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.88)
- From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models (ICML 2025 · similarity 0.88)
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (ICML 2025 · similarity 0.88)
- Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
