Research map / Vision-language and multimodal / Multimodal LLMs
MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MultiPriv, a bilingual (EN/ZH) benchmark with synthetic individual profiles and nine tasks for measuring VLM individual-level privacy perception and reasoning, evaluating 50+ open-source and commercial VLMs.
Problem
placeholder
Approach
Frames privacy risk as a two-stage Privacy Perception and Reasoning (PPR) process and builds a benchmark of synthetic individuals whose identifiers (faces, names, IDs) are linked to sensitive attributes, so identity linkage can be tested with nine VQA-style tasks (single-step, re-ID, chained, cross-modal).
Key results
- Qwen3-VL-32B-Thinking reaches an overall privacy reasoning risk of 0.874, and Gemini-2.5-Flash 0.825 among commercial models.
- QvQ-72B-Preview achieves perception risk of 0.454 and reasoning risk of 0.632, while Gemini-2.5-Pro has the highest perception risk (0.579) with bounding IoU of 0.52.
- GPT-5 refuses 45.60% of tasks yet achieves an answered-only reasoning accuracy of 0.91, versus Qwen3-VL with near-zero refusal (0.01) and 0.88 answered-only accuracy.
Surprising findings
- Strong refusal behavior can mask substantial underlying privacy reasoning capability (GPT-5 answered-only accuracy 0.91 vs overall 0.46).
- Explicit step-by-step or CoT prompts increase privacy leakage for standard VLMs, e.g., GPT-5 reasoning score rises from 0.52 to 0.57.
Limitations
- Uses synthetic profiles, which may not capture noisier real-world privacy scenarios.
- Controlled benchmark scale with 40 individual profiles and 10 images each.
- Threat model limited to visual and cross-modal identity linkage; excludes text-only leakage and external retrieval.
Experimental setup
Datasets: MultiPriv (synthetic profiles and attribute images) WIDER FACE MultiTrust RTVLM PII External Dataset Open-i Medical Image Dataset
Benchmarks: MultiPriv
Metrics: F1 information extraction accuracy (exact match) mean IoU mean accuracy average chain completion percentage task completion accuracy refusal rate answered-only accuracy
Baselines: GPT-4o GPT-5 Gemini-2.5-Pro Gemini-2.5-Flash Claude-Sonnet-4 Qwen3-VL InternVL3.5 Llama-4 QvQ-72B-Preview LLaVA-v1.6 InstructBLIP DeepSeek-VL2
Builds on: chain-of-thought prompting VISPR VizWiz
Closest papers by meaning
- Private Attribute Inference from Images with Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.88)
- PerceptAnon: Exploring the Human Perception of Image Anonymization Beyond Pseudonymization for GDPR (ICML 2024 · similarity 0.88)
- MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.88)
- Disrupting Hierarchical Reasoning: Adversarial Protection for Geographic Privacy in Multimodal Reasoning Models (ICLR 2026 · similarity 0.88)
- Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.