Research map / Large language models / Agents, code and math
Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CI-CoT prompting and CI-RL (GRPO with a keyword-based rule reward) trained on a ~700-example synthetic contextual integrity dataset, showing reduced inappropriate disclosure that transfers to PrivacyLens.
Problem
LLM agents often disclose information that is inappropriate for the task context, violating contextual integrity (CI); current models lack reliable CI judgment.
Approach
Prompt models to reason explicitly about CI before disclosure (CI-CoT), then post-train with GRPO using a rule-based reward that rewards required keywords and penalizes restricted keywords (CI-RL), using a synthetic GPT-4-generated dataset.
Key results
- On the synthetic test set, CI-RL raised Integrity for Qwen2.5-7B-Instruct from 46.9% to 75.0% and Complete from 29.7% to 48.4%.
- CI-RL Qwen2.5-7B-Instruct (Integrity 75.0%, Complete 48.4%) outperformed the baseline Qwen2.5-14B-Instruct (51.6%, 37.5%).
- On PrivacyLens, Qwen2.5-7B-IT with CI-RL reduced leakage rate from 50.3% (CI-CoT) to 33.7%, with helpfulness 2.08.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CI-RL | Qwen2.5-1.5B-IT · Synthetic test | Integrity: 37.5 -> 59.4 | Helped |
| apply CI-RL | Qwen2.5-3B-IT · Synthetic test | Integrity: 31.2 -> 57.8 | Helped |
| apply CI-RL | Qwen2.5-7B-IT · Synthetic test | Integrity: 46.9 -> 75.0 | Helped |
| apply CI-RL | Mistral-7B-IT · Synthetic test | Integrity: 38.8 -> 89.1 | Helped |
| apply CI-RL | Llama-3.1-8B-IT · Synthetic test | Integrity: 61.9 -> 79.7 | Helped |
| apply CI-RL | Qwen2.5-14B-IT · Synthetic test | Integrity: 51.6 -> 78.1 | Helped |
| apply CI-RL | Qwen2.5-7B-IT · Synthetic test | Complete: 29.7 -> 48.4 | Helped |
| apply CI-RL | Mistral-7B-IT · Synthetic test | Complete: 24.5 -> 73.4 | Helped |
| apply CI-RL | Llama-3.1-8B-IT · Synthetic test | Complete: 38.1 -> 62.5 | Helped |
| apply CI-RL | Qwen2.5-7B-IT · PrivacyLens | Leakage rate: 50.3 -> 33.7 | Helped |
| add CI-CoT prompting | Qwen2.5-7B-IT · PrivacyLens | Leakage rate: 50.3 -> 44.8 | Helped |
| add CI-CoT prompting | Qwen2.5-7B-IT · PrivacyLens | Helpfulness: 1.99 -> 2.13 | Helped |
Surprising findings
- Distilled reasoning models (DeepSeek-R1-Distill) underperformed their instruction-tuned bases on the CI task after CI-RL.
- CI-CoT prompting alone reduced leakage on PrivacyLens for some models but lowered helpfulness, showing a safety-helpfulness trade-off.
Limitations
- Relies on a small synthetic dataset with keyword-based rewards that are coarse and string-match based.
- No error bars or multiple runs reported.
- Only tested on models up to 14B; human-annotated CI data is scarce.
Experimental setup
Datasets: ContextualIntegritySyntheticDataset (synthetic, GPT-4 generated) PrivacyLens ConfAIde
Benchmarks: ContextualIntegritySyntheticDataset test split PrivacyLens ConfAIde
Metrics: Integrity Utility Complete leakage rate adjusted leakage rate helpfulness score
Baselines: Qwen2.5-1.5B-Instruct Qwen2.5-3B-Instruct Qwen2.5-7B-Instruct Qwen2.5-14B-Instruct Llama-3.1-8B-Instruct Mistral-7B-Instruct-v0.3 DeepSeek-R1-Distill-Llama-8B DeepSeek-R1-Distill-Qwen-14B Claude 3.7 Sonnet Claude 3.7 Sonnet Thinking Gemini 2.5 Flash Gemini 2.5 Pro
Builds on: GRPO PPO DeepSeek-R1 VERL PrivacyLens
Closest papers by meaning
- It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs (NeurIPS 2026 · similarity 0.93)
- Contextualized Privacy Defense for LLM Agents (ICML 2026 · similarity 0.90)
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.90)
- Privacy Reasoning in Ambiguous Contexts (NeurIPS 2025 · similarity 0.88)
- PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training (ICML 2026 · similarity 0.88)
- Position: Contextual Integrity is Inadequately Applied to Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.