Research map / Vision-language and multimodal / Cross-modal learning
Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models
Yuriel Ryan, Ip Hei Man, Adriel Kuek, Paul Pu Liang, Roy Ka-Wei Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VLM-generated captions to the text modality | Hateful Memes training data · Hateful Memes train | redundancy R: 0.0553 -> 0.2319 (+319%) | Improved |
| add VLM-generated captions to the text modality | Hateful Memes training data · Hateful Memes train | unique visual U_V: 0.3465 -> 0.1710 (-51%) | Improved |
| use meaningful captions instead of random text (ablation) | interaction transfer · Hateful Memes test | redundancy R: 0.0653 -> 0.0939 | Improved |
| use larger captioner (Qwen2.5 32B) instead of SmolVLM-2B (ablation) | interaction transfer · Hateful Memes test | redundancy R: 0.0752 -> 0.0939 | Improved |
| caption videos in a synergy-dominant dataset | UR-FUNNY · UR-FUNNY test | unique text U_T: 0.0004 -> 0.0034 (+750.0%) | Hurt |
| caption videos in a synergy-dominant dataset | UR-FUNNY · UR-FUNNY test | synergy S: 0.0043 -> 0.0030 (-30.2%) | Hurt |
| add 50% extra redundancy via self-captioning | SmolVLM · macro average | visual-induced errors: -38.3% | Improved |
| add 50% extra redundancy via self-captioning | SmolVLM · macro average | accuracy and consistency: +4.0% accuracy, +16.8% consistency | Improved |
| add 50% extra redundancy via self-captioning | SmolVLM · macro average | language-induced and mixed errors: +15.2% LI, +42.9% Mix | Hurt |
| add 25% extra redundancy via self-captioning | LLaVA-OneVision · macro average | visual-induced errors: -34.4% | Improved |
| add 25-50% redundancy | SmolVLM and LLaVA-OneVision · corruption severity | performance stability | Improved |
| caption 100% of training samples (ablation) | SmolVLM-2B · hate speech detection | performance stability: DeltaP approx -4.07 | Hurt |
Closest papers by meaning
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.88)
- Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.88)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.88)
- Calibrated Self-Rewarding Vision Language Models (NeurIPS 2024 · similarity 0.87)
- Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework (CVPR 2026 · similarity 0.87)
- Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
