Research map / Large language models / Agents, code and math
Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CoDI-Eval, a benchmark that evaluates LLMs' controllable text generation under diversified natural-language instructions across sentiment, topic, multi-aspect, length, keyword, and toxicity-avoidance tasks, using GPT-3.5-turbo to expand and rewrite instructions and automatic classifiers for scoring.
Problem
Existing controllable text generation evaluations use discrete control conditions or fixed templates and do not test whether instruction-following LLMs honor explicit constraints expressed in varied natural language.
Approach
Build a large instruction set by LLM-driven expansion of seed instructions followed by bootstrapped text-rewriting diversification (basic and GPT-4-derived rewrite styles), then score responses automatically with task-specific classifiers, rule-based checks, and Perspective API.
Key results
- GPT-4 (0613) achieves the highest zero-shot average accuracy of 84.82%, while the best open-source model trails commercial models by over 10 points.
- Length control is the hardest hard constraint: GPT-3.5-turbo reaches only 55% on length tasks, versus 88% on sentiment.
- Few-shot prompting does not reliably improve controllability; it lowers average accuracy for several models and reduces response diversity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the simple few-shot prompt instead of zero-shot (ablation) | GPT-3.5-turbo (0613) · CoDI-Eval overall | average accuracy (%): 80.5 -> 79.33 | Hurt |
| Use the simple few-shot prompt instead of zero-shot (ablation) | Qwen-72B-chat · CoDI-Eval overall | average accuracy (%): 72.73 -> 70.21 | Hurt |
| Use the simple few-shot prompt instead of zero-shot (ablation) | LLaMA2-7B-chat · CoDI-Eval overall | average accuracy (%): 67.57 -> 68.38 | Helped |
| Use the simple few-shot prompt instead of zero-shot (ablation) | WizardLM-13B-V1.2 · CoDI-Eval overall | average accuracy (%): 69.4 -> 69.51 | No clear effect |
| Use random few-shot demonstrations (ablation) | GPT-3.5-turbo (0613), Length equal-to subtask · CoDI-Eval Length, equal to | accuracy: 8% accuracy ... 4% reduction | Hurt |
| Select demonstrations whose text length matches or is close to the target requirement (ablation) | GPT-3.5-turbo (0613), Length equal-to subtask · CoDI-Eval Length, equal to | accuracy: 8% accuracy ... 7% improvement | Helped |
| Add instruction diversification after expansion (ablation) | CoDI-Eval instruction set · Instruction diversity | Rouge-L similarity | Helped |
| Fine-tune on length-control instructions for 10 words (ablation) | LLaMA-7B · Length control generation | accuracy: 20% | Helped |
| Fine-tune on length-control instructions for 200 words (ablation) | LLaMA-7B · Length control generation | accuracy: 1% | Helped |
| Lower temperature from 1.0 to 0.1 (ablation) | GPT-3.5-turbo (0613) · CoDI-Eval overall | average accuracy (%): 80.5 -> 77.96 | Hurt |
Surprising findings
- Simple few-shot in-context demonstrations often fail to improve, and sometimes hurt, controllable generation accuracy.
- Fine-tuning LLaMA-7B on length-control instructions yields limited generalization, suggesting length constraints are hard to learn.
Limitations
- Automatic evaluators (sentiment/topic classifiers) are only partially validated against humans, with 89.5% to 98.5% consistency.
- Attributing failures to misunderstanding versus failure to execute the instruction is not resolved.
- Instructions are generated and rewritten by GPT-3.5-turbo, which may introduce generator-specific biases.
Experimental setup
Datasets: CommonGen TweetTopic RealToxicPrompts CNNDM ShareGPT
Benchmarks: CoDI-Eval
Metrics: accuracy self-BLEU Rouge-L Vendi score
Baselines: InstructCTG Bound-Cap-LLM GPT-4 Qwen-72B-chat Vicuna-13B LLaMA2-13B-chat
Builds on: InstructCTG Self-Instruct PPLM DExperts
Closest papers by meaning
- Adaptable Logical Control for Large Language Models (NeurIPS 2024 · similarity 0.86)
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.85)
- LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions (NeurIPS 2025 · similarity 0.85)
- Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.84)
- Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo (ICLR 2025 · similarity 0.84)
- Teaching Language Models to Critique via Reinforcement Learning (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.