Research map / Large language models / Instruction and fine-tuning
Analyzing Finetuning Representation Shift for Multimodal LLMs Steering
Pegah Khayatan, Mustafa Shukor, Jayneel Parekh, Arnaud Dapogny, Matthieu Cord
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Yes->No steering vector at last layer | LLaVA-1.5 steering · VQAv2 val subset | target answer count: +828 | Improved |
| apply Yes->No steering vector | LLaVA-1.5 steering · VQAv2 yes/no accuracy | accuracy (%): 90.82 -> 69.03 | Hurt |
| apply White->Black steering vector | LLaVA-1.5 steering · VQAv2 other accuracy | accuracy (%): 71.10 -> 58.36 | Hurt |
| coarse steering toward yes/no, number, other answer types | LLaVA-1.5 · VQAv2 answer-type counts | target type count: 366 -> 557; 122 -> 201; 494 -> 501 | Improved |
| steer captions toward colors | LLaVA-1.5 steering · COCO caption style | caption count: 1309 -> 2561 | Improved |
| use fine-grained instead of coarse steering for gender debiasing | LLaVA-1.5 steering · COCO gender debiasing | captions converted: 232 -> 632 | Improved |
| use fine-grained instead of coarse steering for gender debiasing | Qwen2-VL-Instruct steering · COCO gender debiasing | captions converted: 134 -> 300 | Improved |
| apply safety steering vector | Qwen2-VL-Instruct · MM-SafetyBench subset | ASR: 45/100 -> 5/100 | Improved |
| add shift magnitude alpha=1 for concept shift (ablation) | concept shift vectors · VG fine-tunings concept recovery | T-Overlap | Improved |
| increase number of concepts K (ablation) | concept recovery · concept recovery T-Overlap | T-Overlap | No clear effect |
| extract concepts from deeper layers (ablation) | concept recovery · concept recovery T-Overlap | T-Overlap gap | Improved |
| steer at deeper layers (ablation) | LLaVA-1.5 steering · VQAv2 | target answer count | Improved |
Closest papers by meaning
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.88)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.88)
- Re-Imagining Multimodal Instruction Tuning: A Representation View (ICLR 2025 · similarity 0.88)
- A Concept-Based Explainability Framework for Large Multimodal Models (NeurIPS 2024 · similarity 0.87)
- Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
