Research map / Large language models / AI and society
metaTextGrad: Automatically optimizing language model optimizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes metaTextGrad, a meta-optimizer that tunes the prompts and combines the structures of LLM-based optimizers (TextGrad TGD, ADAS-TG) so they align with a specific task.
Problem
LLM-based optimizers used for prompt and program optimization are hand-designed, general-purpose and fixed, so they are not aligned with specific tasks and cannot be combined adaptively.
Approach
Treat the optimizer itself as an object to optimize: a meta prompt optimizer refines each optimizer's meta-prompt using task examples, and a meta structure optimizer (an o1 LLM) merges multiple prompt-optimized optimizers into a composite, with both selected by validation score in an outer loop.
Key results
- Average test accuracy of 0.53 with GPT-4o-mini across four benchmarks, versus 0.47 for the best baseline (zero-shot MIPROv2) and 0.45 for ADAS-TG.
- On BBH Dyck Languages, test accuracy rises from 0.16 (ADAS-TG) to 0.37 (metaTextGrad).
- Optimizers meta-optimized on GPT-4o-mini transfer to Claude 3 Haiku on Dyck Languages (0.35 test vs 0.34 for ADAS-TG).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply meta-optimization to the optimizer | metaTextGrad vs ADAS-TG · BBH Word Sorting test | accuracy: 0.58 -> 0.65 | Helped |
| apply meta-optimization to the optimizer | metaTextGrad vs 8-shot MIPROv2 · BBH Dyck Languages test | accuracy: 0.26 -> 0.37 | Helped |
| apply meta-optimization to the optimizer | metaTextGrad vs Self-consistency (8) · GPQA Diamond test | accuracy: 0.42 -> 0.40 | Hurt |
| apply meta-optimization to the optimizer | metaTextGrad vs Zero-shot MIPROv2 · MMLU Abstract Algebra test | accuracy: 0.77 -> 0.71 | Hurt |
| add meta prompt optimization to TGD (ablation) | TGD Optimizer · BBH Dyck Languages test | accuracy: 0.10 -> 0.24 | Helped |
| add meta prompt optimization to ADAS-TG (ablation) | ADAS-TG · BBH Dyck Languages test | accuracy: 0.16 -> 0.37 | Helped |
| add meta structure optimization (ablation) | ADAS-TG · BBH Dyck Languages test | accuracy: 0.16 -> 0.16 | No clear effect |
| transfer the optimizer trained on GPT-4o-mini to Claude 3 Haiku | metaTextGrad · BBH Dyck Languages test | accuracy: 0.34 -> 0.35 | Helped |
| transfer the optimizer trained on GPQA Diamond to Abstract Algebra | metaTextGrad · MMLU Abstract Algebra test | accuracy: 0.71 -> 0.77 | Helped |
| use metaTextGrad rather than zero-shot CoT with GPT-4o-mini | BBH Dyck Languages · GPT-4o-mini | performance: 0.05 -> 0.37 | Helped |
Surprising findings
- Optimizer baselines on MMLU and GPQA Diamond are judged by GPT-4o because DSPy outputs often do not follow the required answer format, so exact-match scoring was not applicable.
- The meta structure optimizer found no better option on Dyck Languages, so the final metaTextGrad result equals the prompt-optimized ADAS-TG.
Limitations
- Very small train/validation sets (10 to 50 examples per task), with results averaged over five seeds, so variance and overfitting risk are high.
- The meta-optimizer itself is hand-designed rather than learned, as the authors acknowledge.
- Gains are uneven across benchmarks; metaTextGrad does not beat the best baseline on GPQA Diamond test (0.40 vs 0.42 for self-consistency).
Experimental setup
Datasets: BBH Word Sorting BBH Dyck Languages MMLU Abstract Algebra GPQA Diamond
Benchmarks: BBH Word Sorting BBH Dyck Languages GPQA Diamond MMLU Abstract Algebra
Metrics: accuracy
Baselines: Zero-shot CoT 8-shot CoT Self-consistency Best of N MIPROv2 TextGrad TGD ADAS-TG
Builds on: TextGrad DSPy ADAS MIPRO OPRO
Closest papers by meaning
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.90)
- Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization (ICML 2025 · similarity 0.89)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- Large Language Models as Optimizers (ICLR 2024 · similarity 0.88)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.88)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.