Research map / Adversarial robustness and security / Attacks on text and graphs
Do Vision-Language Models Leak What They Learn? Adaptive Token-Weighted Model Inversion Attacks
Ngoc-Bao Nguyen, Sy-Tuyen Ho, Koh Jun Hao, Ngai-Man Cheung
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| aggregate losses over the whole output sequence instead of per-token updates (SMI vs TMI) (ablation) | model inversion on LLaVA-v1.6-7B · FaceScrub | AttAccM: 44.34% -> 59.17% (LLOM) | Improved |
| add adaptive token weighting by visual attention (SMI-AW vs SMI) (ablation) | SMI · FaceScrub | AttAccM: 59.17% -> 61.01% (LLOM) | Improved |
| add adaptive token weighting by visual attention (SMI-AW vs SMI) (ablation) | SMI · FaceScrub | AttAccM: 40.97% -> 41.16% (LCE) | Improved |
| add adaptive token weighting by visual attention (SMI-AW vs SMI) (ablation) | SMI · FaceScrub | AttAccD Top1: 33.47% -> 37.62% (LLOM) | Improved |
| add adaptive token weighting by visual attention (SMI-AW vs SMI) (ablation) | SMI · CelebA | AttAccD Top1: 38.30% -> 45.25% | Improved |
| add adaptive token weighting by visual attention (SMI-AW vs SMI) (ablation) | SMI · StanfordDogs | AttAccM: 75.94% -> 78.13% | Improved |
| add adaptive token weighting by visual attention (SMI-AW vs SMI) (ablation) | SMI · StanfordDogs | delta_eval: 76.98 -> 81.66 | Hurt |
| update latent K times per token before moving on (TMI-C vs TMI) (ablation) | TMI · FaceScrub | AttAccM: 44.34% -> 31.16% (LLOM) | Hurt |
| use max-margin loss instead of cross-entropy in SMI (ablation) | SMI · FaceScrub | AttAccM: 40.97% -> 55.52% | Improved |
| use logit-maximization loss instead of cross-entropy in SMI-AW (ablation) | SMI-AW · FaceScrub | AttAccM: 41.16% -> 61.01% | Improved |
| switch target VLM from LLaVA-v1.6-7B to Qwen2.5-VL-7B | SMI-AW (LLOM) · FaceScrub | AttAccM: 61.01% -> 32.03% | Hurt |
| evaluate with human raters instead of MLLM judge | SMI-AW (LLOM) · FaceScrub | attack accuracy: 61.01% (AttAccM) vs 56.93% (AttAccH) | Mixed |
Closest papers by meaning
- Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.90)
- Membership Inference Attacks against Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update (AAAI 2025 · similarity 0.88)
- Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.88)
- MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.88)
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
