Research map / Large language models / Language, safety and interpretability
COLLIE: Systematic Construction of Constrained Text Generation Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces COLLIE, a grammar-based framework that specifies compositional text constraints, auto-extracts instances from corpora, and renders/evaluates them, releasing the COLLIE-v1 dataset (2,080 instances, 13 structures) and evaluating five instruction-tuned LLMs.
Problem
Existing constrained text generation benchmarks focus on fixed, easy constraint types and require tailored data pipelines, so they fail to evaluate compositional constraints that challenge modern LLMs.
Approach
A context-free grammar specifies constraint structures over generation levels (char to passage) with count and position base-constraints combined by logical and/or; a compiler extracts constraint values from real corpora (Wikipedia, CC-News, Project Gutenberg), renders rule-based instructions, and automatically checks outputs.
Key results
- GPT-4 achieves an average constraint satisfaction rate of 50.9% zero-shot on COLLIE-v1, more than twice the rate of non-GPT models.
- GPT-4 average pass@20 is above 63%, versus 32% for GPT-3.5.
- Natural language feedback over rounds lifts GPT-4 performance by about 20% after one feedback round, plateauing near 66%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-shot prompting instead of zero-shot prompting (ablation) | GPT-3.5 · COLLIE-v1 | constraint satisfaction rate: 23.1 -> 23.6 | No clear effect |
| Use one-shot prompting instead of zero-shot prompting (ablation) | GPT-4 · COLLIE-v1 | constraint satisfaction rate: 40.7 -> 39.4 | No clear effect |
| Add automated natural-language feedback between generation rounds | GPT-4 · COLLIE-v1 | constraint satisfaction rate: 20% improvement | Helped |
| Continue feedback for three additional rounds after the second round | GPT-4 · COLLIE-v1 | constraint satisfaction rate: 66% | No clear effect |
Surprising findings
- Position matters: GPT-4 reaches 100% on starting a sentence with a given word but only 40-60% on ending a sentence with a given word.
- One-shot prompting with an example gives results similar to zero-shot for GPT-3.5 and GPT-4, suggesting difficulty lies in generation under constraints rather than understanding them.
Limitations
- Automated filtering and post-processing are imperfect and can affect constraint quality.
- Representative constraint structures cannot capture all dimensions of model capability.
- Grammar currently covers only count and position base-constraints; semantic constraints like part of speech or sentiment are left to future work.
Experimental setup
Datasets: Wikipedia (20220301.en) CC-News Project Gutenberg (Gutenberg, dammit)
Benchmarks: COLLIE-v1
Metrics: constraint satisfaction rate pass@k word validity passage coherence score
Baselines: GPT-3.5 PaLM-2 Vicuna-7B Alpaca-7B
Builds on: nltk sentence tokenizer
Closest papers by meaning
- Guaranteed Generation from Large Language Models (ICLR 2025 · similarity 0.84)
- Retrieval is Accurate Generation (ICLR 2024 · similarity 0.84)
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.84)
- Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.84)
- Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation (ICML 2024 · similarity 0.83)
- Beyond Prompt Engineering: A Reinforced Token-Level Input Refinement for Large Language Models (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.