Research map / Large language models / Language, safety and interpretability
CriSPO: Multi-Aspect Critique-Suggestion-guided Automatic Prompt Optimization for Text Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CriSPO, an automatic prompt optimization method for text generation that uses multi-aspect LLM critiques and suggestions to guide an OPRO-style optimizer, plus an Automatic Suffix Tuning extension for multi-metric optimization.
Problem
Automatic prompt engineering methods are built for discriminative tasks with a single metric, giving weak optimization signals for text generation, which has multiple aspects and metrics.
Approach
A critique-suggestion meta-prompt discovers task-specific aspects, critiques generated vs. reference outputs, and writes edit suggestions; a receptive optimizer consumes the enriched trajectory (prompts, scores, critiques, suggestions) with chain-of-thought and a flexible template with placeholders. AST optimizes a suffix appended to a fixed prompt to improve a secondary metric (AlignScore) while
Key results
- On summarization, CriSPO improves average ROUGE-1 over OPRO by about 3-4 points across models (e.g., Claude Instant avg R1 44.7 vs OPRO 40.4).
- On SQuAD with Claude Instant, CriSPO reaches 87.8 F1 vs 78.5 for OPRO and 79.5 for 0-shot manual.
- AST improves AlignScore on average by 4.3 points (CNN +11.7) with ROUGE-1 roughly unchanged (44.9 vs 45.1 seed).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace OPRO with CriSPO | automatic prompt optimization · Summarization average, Claude Instant | ROUGE-1: 40.4 -> 44.7 | Helped |
| replace OPRO with CriSPO | automatic prompt optimization · Summarization average, Claude3 Sonnet | ROUGE-1: 42.2 -> 46.8 | Helped |
| add 3-shot in-context examples to CriSPO | CriSPO · Summarization average, Claude Instant | ROUGE-1: 44.7 -> 49.0 | Helped |
| replace OPRO with CriSPO | automatic prompt optimization · NarrativeQA, Claude Instant | ROUGE-L: 59.4 -> 75.1 | Helped |
| replace OPRO with CriSPO | automatic prompt optimization · MedMCQA, Claude Instant | accuracy: 50.5 -> 52.3 | Helped |
| add critique-suggestion module (ablation) | CriSPO · SAMSum | ROUGE-1: 42.8 -> 44.4 | Helped |
| add step-by-step CoT optimization (ablation) | CriSPO · SAMSum | ROUGE-1: 43.9 -> 44.4 | Helped |
| add flexible task-prompt template (ablation) | CriSPO · SAMSum | ROUGE-1: 42.2 -> 44.4 | Helped |
| add critique-suggestion and CoT optimization (ablation) | CriSPO · SAMSum | ROUGE-1: 37.4 -> 44.4 | Helped |
| use freely discovered multi-aspects instead of no multi-aspects (ablation) | CriSPO critique-suggestion · SAMSum | ROUGE-1: 41.1 -> 44.4 | Helped |
| replace free multi-aspect discovery with predefined aspects (ablation) | CriSPO critique-suggestion · SAMSum | ROUGE-1: 44.4 -> 44.5 | No clear effect |
| tune an appended suffix for AlignScore while keeping the main prompt fixed | CriSPO with AST · CNN | AlignScore: +11.7 | Helped |
Surprising findings
- Predefined aspects give no significant gain over free-form aspect discovery, suggesting the critique LLM identifies relevant dimensions itself.
- CriSPO outperforms OPRO even on MedMCQA classification, despite being designed for generation.
Limitations
- Costly in LLM API tokens and time, especially for long inputs and RAG settings
- Requires manual prompt-format adaptation for different LLMs; evaluation metrics like ROUGE and AlignScore are imperfect
- Does not outperform state-of-the-art supervised fine-tuning on some datasets
Experimental setup
Datasets: CNN/DailyMail MeetingBank SAMSum ACI-Bench Natural Questions TriviaQA SQuAD NarrativeQA MedMCQA
Benchmarks: CNN/DailyMail MeetingBank SAMSum ACI-Bench Natural Questions TriviaQA SQuAD NarrativeQA MedMCQA
Metrics: ROUGE-1 ROUGE-2 ROUGE-L AlignScore BERTScore Exact Match F1 accuracy human normalized recall
Baselines: Manual 0-shot Manual 3-shot OPRO Full prompt tuning
Builds on: OPRO APE Reflexion
Closest papers by meaning
- The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.86)
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation (ICML 2024 · similarity 0.86)
- Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing (ICLR 2024 · similarity 0.86)
- Beyond Prompt Engineering: A Reinforced Token-Level Input Refinement for Large Language Models (AAAI 2025 · similarity 0.86)
- Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter (AAAI 2025 · similarity 0.85)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.