Research map / Vision-language and multimodal / Multimodal LLMs
DIS-CO: Discovering Copyrighted Content in VLMs Training Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIS-CO, which detects whether copyrighted content (movie frames) was in a VLM's training data by eliciting free-form title predictions from frames, and introduces the MovieTection benchmark (14,000 frames from 100 movies).
Problem
Verifying whether copyrighted content was used to train a vision-language model without access to its training data, including black-box models where token logits are unavailable.
Approach
Query a VLM repeatedly with frames from a target movie and ask for the title in free-form generation (no answer options), removing answers also obtained from captions alone, and compare accuracy on suspect movies against clean (post-cutoff) movies to infer memorization of visual content.
Key results
- On neutral frames, DIS-CO achieves average AUC 0.930 versus 0.778 for MCQA, 0.858 for captions, and 0.481 for Rényi (average over four models).
- GPT-4o reaches 0.338 accuracy on suspect movies' neutral frames versus 0.002 on clean movies with DIS-CO.
- Humans recognize seen movies' neutral frames at 0.19 accuracy but only 0.02 on unseen titles, suggesting VLM performance is not explained by generalization alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIS-CO instead of the strongest baseline, Captions | MovieTection detection · MovieTection neutral frames | AUC: 0.858 -> 0.930 | Helped |
| Use DIS-CO instead of the strongest baseline, Captions | MovieTection detection · MovieTection main frames | AUC: 0.949 -> 0.988 | Helped |
| Remove correct frame predictions overlapping with captions (ablation) | DIS-CO · MovieTection neutral frames | AUC: 0.930 -> 0.928 | No clear effect |
| Remove correct frame predictions overlapping with captions (ablation) | DIS-CO · MovieTection main frames | AUC: 0.988 -> 0.983 | Hurt |
| Use free-form image-to-title prediction instead of captions | GPT-4o · MovieTection neutral frames, suspect movies | accuracy: 0.128 -> 0.338 | Helped |
| Use free-form image-to-title prediction instead of MCQA | GPT-4o · MovieTection neutral frames, clean movies | accuracy: 0.410 -> 0.002 | Hurt |
| Reduce input resolution from 1126 × 512 to 563 × 256 (ablation) | DIS-CO with GPT-4o · MovieTection five-movie subset | accuracy: 0.79 -> 0.74 | Hurt |
| Reduce input resolution from 1126 × 512 to 282 × 128 (ablation) | DIS-CO with GPT-4o · MovieTection five-movie subset | accuracy: 0.79 -> 0.67 | Hurt |
| Add helpful cues to the prompt (ablation) | DIS-CO with GPT-4o · MovieTection five-movie subset | accuracy: 0.79 -> 0.89 | Helped |
| Paraphrase the neutral default prompt (ablation) | DIS-CO with GPT-4o · MovieTection five-movie subset | accuracy: 0.79 -> 0.79 | No clear effect |
| Fine-tune on Moana frames with the label “Copyrighted Content” | Qwen2-VL 7B · Frozen neutral frames | accuracy: 0.000 -> 0.450 | Helped |
| Fine-tune on Moana frames with the label “Copyrighted Content” | Qwen2-VL 7B · Lion King neutral frames | accuracy: 0.000 -> 0.625 | Helped |
Surprising findings
- MCQA accuracy on suspect movies is high (0.721 for GPT-4o) but also yields many false positives on clean movies (0.410), inflating its accuracy.
- GPT-4o acknowledges 20 of 50 post-cutoff movies, though with near-zero accuracy on them.
Limitations
- Free-form generation may miss memorization that does not surface as verbatim title recall, so negatives are not conclusive.
- Movie-level detection on 100 movies; relies on box-office and temporal proxies that can confound membership with popularity and time effects.
- Models' knowledge from non-training sources (posters, trailers) can produce correct answers; the human baseline only partly controls for this.
Experimental setup
Datasets: MovieTection VL-MIA/Flickr COCO custom comic book pages
Benchmarks: MovieTection VL-MIA/Flickr comic book pages
Metrics: AUC accuracy standard deviation over resampling
Baselines: Captions MCQA Rényi (alpha=0.5) / MaxRényi-K%
Builds on: Min-K%-Prob Rényi entropy membership inference (MaxRényi-K%) LoRA LlamaFactory
Closest papers by meaning
- DE-COP: Detecting Copyrighted Content in Language Models Training Data (ICML 2024 · similarity 0.88)
- Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images (ICLR 2025 · similarity 0.87)
- Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content? (AAAI 2026 · similarity 0.87)
- Copyright Traps for Large Language Models (ICML 2024 · similarity 0.85)
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations (ICLR 2026 · similarity 0.85)
- ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.