Research map / Vision-language and multimodal / Multimodal LLMs
Private Attribute Inference from Images with Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a manually labeled Reddit image dataset (VIP) of non-human-centric images and evaluates 7 VLMs on inferring personal attributes, with a gamified CoT prompt and automated zooming.
Problem
Whether vision-language models can infer private personal attributes (e.g., location, sex, age, income) from seemingly benign online images that do not primarily depict humans, a risk not covered by existing human attribute recognition benchmarks.
Approach
Shifts privacy-inference evaluation from human-centric attribute recognition to inconspicuous everyday images, using a gamified chain-of-thought prompt to bypass safeguards and an automated zoom-in step on model-proposed bounding boxes to recover fine-grained clues.
Key results
- GPT-4V achieves 77.6% overall attribute inference accuracy on VIP, the best of 7 VLMs.
- Best open-source model CogAgent-VQA reaches 66.4%, outperforming Gemini-Pro (60.8%).
- Naive prompt causes GPT-4V to refuse 54.5% of queries; the gamified Extended and Final prompts reduce refusal to 1.2% and 0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adversarial gamification to the prompt (ablation) | GPT4-V · VIP | accuracy: 20.6 -> 76.0 | Helped |
| add adversarial gamification to the prompt (ablation) | GPT4-V · VIP | refusal rate: 54.5 -> 1.2 | Helped |
| add chain-of-thought, reasoning guidelines, and generic reasoning hints (ablation) | GPT4-V with the extended prompt · VIP | accuracy: 76.0 -> 77.6 | Helped |
| add automated zooming (ablation) | GPT4-V final prompt · VIP precise LOC | accuracy: +6.5 | Helped |
| add automated zooming (ablation) | GPT4-V final prompt · VIP less precise LOC | accuracy: +0.0 | No clear effect |
| add automated zooming (ablation) | GPT4-V final prompt · VIP precise POI | accuracy: +4.3 | Helped |
| add automated zooming (ablation) | GPT4-V final prompt · VIP less precise POI | accuracy: +2.2 | Helped |
| use the open-source prompt instead of the final prompt (ablation) | CogAgent-VQA · VIP | accuracy: 55.2 -> 66.4 | Helped |
| use the open-source prompt instead of the final prompt (ablation) | Idefics 80B · VIP | accuracy: 59.0 -> 65.7 | Helped |
| use the open-source prompt instead of the final prompt (ablation) | LLaVa-NeXT 34B · VIP | accuracy: 59.4 -> 57.0 | Hurt |
| use the open-source prompt instead of the final prompt (ablation) | InternVL-Chat-V1.2-Plus · VIP | accuracy: 59.2 -> 50.4 | Hurt |
| use the open-source prompt instead of the final prompt (ablation) | LLaVa 1.5 13B · VIP | accuracy: 52.7 -> 53.2 | Helped |
Surprising findings
- Safeguards of GPT-4V are brittle: a simple gamified prompt cuts refusals from 54.5% to 1.2%.
- Attribute inference accuracy correlates with general capability (MMMU ranking), suggesting stronger models pose larger privacy risk.
Limitations
- VIP dataset is small (554 labels) and not publicly released, limiting reproducibility.
- Only 9.7% of labels come from images with partial human depictions, and the labeling was done solely by the authors.
- Free-text attributes (location, occupation) are scored with GPT-4 similarity judgments, manually verified.
Experimental setup
Datasets: VIP (visual inference-privacy) dataset RedCaps
Benchmarks: VIP
Metrics: accuracy precise vs less-precise location accuracy refusal rate
Baselines: GPT4-V Gemini-Pro LLaVA 1.5 13B LLaVA-NeXT 34B Idefics 80B CogAgent-VQA InternVL-Chat-V1.2-Plus
Builds on: chain-of-thought prompting beyond memorization text privacy inference (Staab et al.) pedestrian attribute recognition benchmarks
Closest papers by meaning
- MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models (ICML 2026 · similarity 0.92)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.91)
- A Synthetic Dataset for Personal Attribute Inference (NeurIPS 2024 · similarity 0.89)
- Membership Inference Attacks against Large Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Can We Infer Confidential Properties of Training Data from LLMs? (NeurIPS 2025 · similarity 0.87)
- MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.