Research map / Large language models / Agents, code and math
INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes INDICT, a framework where a safety-driven critic and a helpfulness-driven critic interact in an internal dialogue, grounded with external tools, to critique and revise LLM-generated code at both generation (preemptive) and execution (post-hoc) stages.
Problem
LLMs for code often produce insecure code or fail to balance helpfulness and safety, especially under complex or malicious instructions, and fine-tuning for safety is costly and not scalable.
Approach
Two separately prompted critics (safety and helpfulness) converse autonomously, summarize their dialogue, and use tool-grounded code search and code review actions; their critiques guide an actor LLM both before execution (preemptive) and after execution feedback (post-hoc), with no training.
Key results
- INDICT yields about +10% absolute improvement in safety and helpfulness across all evaluated models per the abstract.
- With GPT-4o-mini on a CyberSecEval-1 validation split, INDICT reaches 90.9% safety and 81.4% helpfulness, above baselines such as Reflexion+ and CAMEL+.
- With CodeLlama-7b-instruct, INDICT on top of CodeUltraFeedback SFT+DPO raises S+H from 60.9 to 73.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add INDICT to direct generation | CodeLlama-7B-Instruct · CyberSecEval-1 validation split (20%) | Safety: 56.3 -> 65.3 | Helped |
| Add INDICT to direct generation | CodeLlama-7B-Instruct · CyberSecEval-1 validation split (20%) | Helpfulness: 50.0 -> 62.1 | Helped |
| Add INDICT to SFT+DPO fine-tuning | CodeLlama-7B-Instruct with CodeUltraFeedback · CyberSecEval-1 validation split (20%) | Safety: 63.9 -> 74.9 | Helped |
| Add INDICT to SFT+DPO fine-tuning | CodeLlama-7B-Instruct with CodeUltraFeedback · CyberSecEval-1 validation split (20%) | Helpfulness: 57.9 -> 72.4 | Helped |
| Use separate safety and helpfulness critics (ablation) | INDICT · CyberSecEval-1 validation split (20%) | Safety: 87.3 -> 90.9 | Helped |
| Use separate safety and helpfulness critics (ablation) | INDICT · CyberSecEval-1 validation split (20%) | Helpfulness: 76.4 -> 81.4 | Helped |
| Add critic summarization instead of passing full dialogue history (ablation) | INDICT · CyberSecEval-1 validation split (20%) | Safety: 87.9 -> 90.9 | Helped |
| Add critic summarization instead of passing full dialogue history (ablation) | INDICT · CyberSecEval-1 validation split (20%) | Helpfulness: 72.2 -> 81.4 | Helped |
| Use thought-action-observation critics instead of RAG-based critics (ablation) | INDICT · CyberSecEval-1 validation split (20%) | Safety: 87.9 -> 90.9 | Helped |
| Use thought-action-observation critics instead of RAG-based critics (ablation) | INDICT · CyberSecEval-1 validation split (20%) | Helpfulness: 74.4 -> 81.4 | Helped |
| Use thought-action-observation critics instead of tool-based critics (ablation) | INDICT · CyberSecEval-1 validation split (20%) | Safety: 85.5 -> 90.9 | Helped |
| Use thought-action-observation critics instead of tool-based critics (ablation) | INDICT · CyberSecEval-1 validation split (20%) | Helpfulness: 72.7 -> 81.4 | Helped |
Surprising findings
- Simply adding safety criteria to existing multi-agent or self-refine baselines is suboptimal, and a single critic covering both criteria performs worse than two decoupled critics.
Limitations
- Relies on instruction-following ability of LLMs to play actor and critic roles; definitions of safety and helpfulness in code are non-trivial.
- About 3-4x time cost versus single-LLM generation.
- Critics may miss the right safety or helpfulness issues; safety beyond code (bias, offensive content) not considered.
Experimental setup
Datasets: CyberSecEval-1 CyberSecEval-2 CVS CAMEL HarmBench
Benchmarks: CyberSecEval-1 CyberSecEval-2 CVS CAMEL HarmBench
Metrics: safety rate (rule-based detector) helpfulness winning rate safety (benign percentage) S+H average
Baselines: Reflexion CAMEL Self-debug Self-repair AgentCoder MetaGPT AgentVerse CodeUltraFeedback Llama2-7b-chat
Builds on: Reflexion CAMEL retrieval-augmented generation ReAct
Closest papers by meaning
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.88)
- Instruction Tuning for Secure Code Generation (ICML 2024 · similarity 0.87)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.87)
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.86)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.86)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.