Research map / Video / Video-language understanding
Vid-SME: Membership Inference Attacks against Large Video Understanding Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vid-SME, a membership inference attack for video understanding LLMs that scores videos by the difference in adaptively parameterized Sharma–Mittal entropy between natural and temporally reversed frame sequences.
Problem
Determining whether a video was used in instruction tuning of video understanding LLMs (VULLMs) is unsolved; existing text/image MLLM membership inference methods generalize poorly to video, with near-zero TPR at low FPR as frame count grows.
Approach
Compute Sharma–Mittal entropy over video-token next-token distributions with per-video adaptive q and r (from motion complexity and illumination variation), and use the gap between natural-order and reversed-order frame entropies as the membership score.
Key results
- On LLaVA-NeXT-Video-34B, Vid-SME improves AUC by 28.3%, accuracy by 18.1%, and TPR@5% FPR by 293% over baselines (per the introduction).
- On Video-XL-CinePile-7B, Vid-SME Min_100%/Mean reaches AUC 0.840 and TPR@5% FPR 0.420, while most baselines have TPR@5% FPR near 0.00-0.01.
- On LLaVA-NeXT-Video-7B, baselines get TPR@5% FPR around 0.001, while Vid-SME keeps TPR@5% FPR above 0.05.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Vid-SME rather than the strongest listed baseline | Video-XL-CinePile-7B, 16 frames · Video-XL-CinePile-7B | AUC: 0.703 -> 0.840 | Helped |
| Use Vid-SME rather than the strongest listed baseline | LLaVA-NeXT-Video-34B, 16 frames · LLaVA-NeXT-Video-34B | AUC: 0.590 -> 0.757 | Helped |
| Add adaptive per-video q/r parameterization (ablation) | Vid-SME-5%, Video-XL-NExT-QA-7B · Video-XL-NExT-QA-7B | attack performance | Helped |
| Add reversed-frame entropy comparison (natural minus reversed) (ablation) | Vid-SME-5%, Video-XL-NExT-QA-7B · Video-XL-NExT-QA-7B | attack performance | Helped |
| Change instruction context among I1, I2, and I3 (ablation) | Vid-SME, LLaVA-NeXT-Video-34B, 8 frames · LLaVA-NeXT-Video-34B | AUC: 0.692; 0.714; 0.729 | No clear effect |
| Use short query text instead of full instruction context (ablation) | Vid-SME, Video-XL-CinePile-7B, 8 frames · Video-XL-CinePile-7B | attack performance | Mixed |
| Increase dataset scale from 2140 to 8560 instances (ablation) | Mean Vid-SME, 8 frames · Video-XL-NExT-QA-7B | TPR@5% FPR: 0.071 -> 0.043 | Hurt |
| Increase dataset scale from 2140 to 8560 instances (ablation) | Mean Vid-SME, 16 frames · Video-XL-NExT-QA-7B | TPR@5% FPR: 0.078 -> 0.075 | Mixed |
| Increase sampled frames from 8 to 16 (ablation) | Mean Vid-SME, dataset scale 4280 · Video-XL-NExT-QA-7B | TPR@5% FPR: 0.064 -> 0.104 | Helped |
| Use motion-blurred or brightness-corrupted video frames (ablation) | Vid-SME · video corruption evaluation | attack performance | Hurt |
Surprising findings
- Baseline MIA methods produce AUC both above and below 0.5 across settings, indicating inconsistent membership signals for video.
- Attacks using only a short query text outperform the full instruction context for small K in the Min-K% selection, though the overall difference is small.
Limitations
- Many VULLMs are trained on proprietary or partially released datasets, so member/non-member sets are approximations using public data.
- Non-member sets may not match the training distribution, so the true severity of privacy risk may be misestimated.
- Attack performance is modest in absolute terms for some settings (e.g., AUC around 0.53-0.55 on Video-XL-NExT-QA-7B).
Experimental setup
Datasets: NExT-QA CinePile MLVU Video-XL training set VDC Video-Instruct-100K
Benchmarks: NExT-QA CinePile MLVU VDC Video-Instruct-100K
Metrics: AUC accuracy TPR@5% FPR
Baselines: Perplexity (Loss) Min-K% Prob Max_Prob_Gap MaxRényi-K% ModRényi Modified Entropy
Builds on: Min-K% ModRényi / MaxRényi-K% Video-XL LongVA
Closest papers by meaning
- Membership Inference Attacks against Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing (NeurIPS 2025 · similarity 0.89)
- Variance-Based Membership Inference Attacks Against Large-Scale Image Captioning Models (CVPR 2025 · similarity 0.87)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.87)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.87)
- VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.