Research map / Multi-agent systems and games / LLM agents
Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks GPT-4o-family models on privacy policy tasks (data practice, choice identification, QA, summarization) and builds a GPT-4o-mini-based privacy policy agent evaluated in a 100-participant user study.
Problem
Privacy policies are long and legally complex, so users rarely comprehend them; the paper asks whether LLMs can both match or beat prior NLP models on privacy-policy tasks and help users understand policies via an agent.
Approach
Replace traditional privacy-policy NLP pipelines with zero/few-shot prompting of GPT-4o-family models, and wrap GPT-4o-mini in a LangChain agent that proactively guides users through policies without requiring specific questions.
Key results
- GPT-4o-mini zero-shot micro-average F1 of 0.67 on data practice identification vs 0.58 for LR on the same categories.
- In the 100-participant user study, the agent group scored 2.6/3 comprehension vs 1.8 for controls (t(98)=6.23, p<0.001, d=1.24) and took 5.5 vs 15.8 minutes.
- GPT-4o-mini (top-10) reached ROUGE-1 0.57 on PolicyQA, above BERT-base at 0.56.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o-mini instead of LR for data practice identification | OPP-115 · OPP-115 | micro-average F1-score: 0.58 -> 0.67 | Helped |
| Use few-shot prompting instead of zero-shot prompting | GPT-4o-mini · Choice Identification | F1-score: 0.83 -> 0.91 | Helped |
| Use top-10 selection | GPT-4o-mini · PolicyQA test | ROUGE-1: 0.44 -> 0.57 | Helped |
| Use top-10 selection | GPT-4o-mini · PolicyQA test | ROUGE-2: 0.37 -> 0.50 | Helped |
| Use top-10 selection | GPT-4o-mini · PolicyQA test | ROUGE-L: 0.43 -> 0.56 | Helped |
| Replace CNN+RF with GPT-4o | Policy summarization · Compression ratio 1/16 | ROUGE-1: 0.431 -> 0.429 | Hurt |
| Assist policy reading with the GPT-4o-mini agent instead of direct reading | User study participants · User study | comprehension mean score (0--3): 1.8 (0.7) -> 2.6 (0.6) | Helped |
| Assist policy reading with the GPT-4o-mini agent instead of direct reading | User study participants · User study | time (minutes): 15.8 (2.5) -> 5.5 (1.8) | Helped |
| Assist policy reading with the GPT-4o-mini agent instead of direct reading | User study participants · User study | task difficulty (0--10): 7.8 (1.1) -> 3.2 (1.0) | Helped |
| Assist policy reading with the GPT-4o-mini agent instead of direct reading | User study participants · User study | confidence in privacy management: 2.6 (0.8) -> 4.4 (0.6) | Helped |
Surprising findings
- GPT-4o-based summarization slightly underperformed supervised domain-specific models on the policy summarization task, per the authors' own table.
- Choice identification zero-shot precision was low (0.74) though recall was high (0.94), requiring few-shot prompting to match the baseline.
Limitations
- Short-term interactions only; no longitudinal evaluation of retention or sustained privacy behavior.
- LLM hallucination and bias risks in agent outputs, with no robust validation mechanism reported.
- Benchmark tables contain inconsistencies and missing baseline values (e.g., Other category without baseline, NaN entries), and prompts in the appendix are empty.
Experimental setup
Datasets: OPP-115 PolicyQA Keymanesh et al. (2020) user agreements dataset
Benchmarks: OPP-115 PolicyQA Keymanesh et al. user agreements
Metrics: precision recall F1-score accuracy ROUGE-1 ROUGE-2 ROUGE-L METEOR comprehension score NASA-TLX cognitive load
Baselines: LR SVM HMM BERT fastText CNN+RF CNN+CF BERT-base
Builds on: LangChain OPP-115 PolicyQA Polisis
Closest papers by meaning
- HelpBench: Assessing the Ability of LLMs to Provide Privacy, Safety, and Security Advice (NeurIPS 2026 · similarity 0.88)
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.87)
- PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action (NeurIPS 2024 · similarity 0.87)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.86)
- PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training (ICML 2026 · similarity 0.86)
- Searching for Privacy Risks in LLM Agents via Simulation (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.