Research map / Adversarial robustness and security / Attacks on text and graphs
Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Parameter Learning Attack (PLA), which builds adversarial trigger images by jointly updating the original LVLM's parameters adversarially during a targeted attack, so triggers persist in fine-tuned derivatives for copyright tracking without modifying the released model.
Problem
Tracking copyright of released large vision-language models when users fine-tune them, without altering the published model's parameters or behavior, since backdoor fingerprinting methods require retraining and adversarial triggers overfit to the original model.
Approach
Construct targeted adversarial trigger images while simultaneously updating the original model's parameters in the opposite direction (min-max adversarial learning), simulating fine-tuning resistance so the trigger generalizes to fine-tuned derivatives; applied post-release with no change to the published weights.
Key results
- PLA achieves average TMR of 55% (LoRA) and 52% (full fine-tuning) across six fine-tuned LLaVA-1.5 derivatives, versus 21%/11% for IF and 22%/16% for RNA.
- Ordinary adversarial attacks achieve only 3% (LoRA) and 2% (full FT) average TMR, indicating overfitting to the original model.
- Trigger images yield 0% TMR on unrelated LVLMs (MiniGPT-4, Qwen2-VL, InternVL2, LLaVA-NEXT, InstructBLIP), suggesting specificity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PLA instead of RNA | LLaVA-1.5-7B copyright tracking · Fine-tuned models, LoRA (Average) | TMR: 22% -> 55% | Helped |
| Use PLA instead of RNA | LLaVA-1.5-7B copyright tracking · Fine-tuned models, full fine-tuning (Average) | TMR: 16% -> 52% | Helped |
| Use PLA rather than ordinary adversarial images | LLaVA-1.5-7B copyright tracking · Fine-tuned models, LoRA (Average) | TMR: 3% -> 55% | Helped |
| Use PLA rather than ordinary adversarial images | LLaVA-1.5-7B copyright tracking · Fine-tuned models, full fine-tuning (Average) | TMR: 2% -> 52% | Helped |
| Use PLA rather than transferable attack methods | LLaVA-1.5-7B copyright tracking · Fine-tuned models, full fine-tuning (Average) | TMR: 2% -> 52% | Helped |
| Apply PLA triggers to unrelated LVLMs | Unrelated LVLMs · MiniGPT-4, QWEN2-VL, InternVL2, LLaVA-NEXT, and InstructBLIP | TMR: 0% | Hurt |
| Use the trigger QA pair “Please stop” / “I'm playing games” instead of “Describe the image” / “I won't tell” | PLA · Fine-tuned models across six datasets | TMR: 72% vs. 40% | Helped |
| Add pixel noise, Gaussian blur, or mean blur to trigger images | PLA triggers · ST-VQA, PaintingF, and ChEBI fine-tuned models | TMR: 64% -> 38%; 64% -> 50%; 64% -> 46% | Hurt |
| Prune model weights | PLA-tracked fine-tuned models · ST-VQA, PaintingF, and ChEBI | TMR: 53% -> 38%; 71% -> 49%; 58% -> 41% | Hurt |
| Perturb model weights | PLA-tracked fine-tuned models · ST-VQA, PaintingF, and ChEBI | TMR: 53% -> 43%; 71% -> 59%; 58% -> 43% | Hurt |
| Increase fine-tuning epochs (ablation) | PLA-tracked models · Fine-tuned models | TMR | Hurt |
| Vary the amount of fine-tuning samples (ablation) | PLA-tracked models · Fine-tuned models | TMR | No clear effect |
Surprising findings
- The IF backdoor fingerprint baseline's fingerprints were completely erased on some fine-tuned models, showing poor robustness when transferred to LVLMs.
- Increasing fine-tuning epochs beyond 4 only slightly reduces TMR, so simply fine-tuning longer does not disable the triggers.
Limitations
- Trigger effectiveness is sensitive to hyperparameters such as model learning rate, perturbation budget, and attack steps (0% TMR at small budgets or few steps).
- Performance is low on some tasks (e.g., MathV360k ~36-40% TMR) and the ordinary tracking rate is only moderate.
- Evaluated mainly on LLaVA-1.5-7B derivatives with five QA pairs and 200 images.
Experimental setup
Datasets: ImageNet 2012 validation V7W ST-VQA TextVQA PaintingForm MathV360k ChEBI-20 RefCOCO Hateful Memes
Benchmarks: V7W ST-VQA TextVQA PaintingForm MathV360k ChEBI-20 RefCOCO Hateful Memes
Metrics: target match rate (TMR)
Baselines: Ordinary adversarial attack IF RNA (random noise attack) MIM DIM CroPA
Builds on: PGD LLaVA-1.5 IF (fingerprinting for LLMs) LoRA
Closest papers by meaning
- Echoes of Ownership: Adversarial-Guided Dual Injection for Copyright Protection in MLLMs (CVPR 2026 · similarity 0.90)
- Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.90)
- Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content? (AAAI 2026 · similarity 0.88)
- Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- DIS-CO: Discovering Copyrighted Content in VLMs Training Data (ICML 2025 · similarity 0.87)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.