Research map / Large language models / Agents, code and math
HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization
Hongzheng Chen, Yingheng Wang, Yaohui Cai, Hins Hu, Jiajie Li, Shirley Huang, Chenhui Deng, Rongjian Liang, Shufeng Kong, Haoxing Ren, Samitha Samaranayake, Carla P Gomes and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add evolutionary framework HSEvo on top of Gemini-2.5-Pro | Gemini-2.5-Pro · HeuriGym | QYI: 0.6170 -> 0.4491 | Hurt |
| add evolutionary framework ReEvo on top of Gemini-2.5-Pro | Gemini-2.5-Pro · HeuriGym | QYI: 0.6170 -> 0.4486 | Hurt |
| add evolutionary framework EoH on top of Gemini-2.5-Pro | Gemini-2.5-Pro · HeuriGym | QYI: 0.6170 -> 0.4492 | Hurt |
| increase few-shot demonstrations from 0 to 5 (10 feedback rounds) (ablation) | Gemini-2.5-Pro · HeuriGym pickup and delivery | QYI: 0.2351 -> 0.4196 | Improved |
| increase demonstrations from 3 to 5 (10 feedback rounds) (ablation) | Gemini-2.5-Pro · HeuriGym pickup and delivery | QYI: 0.2829 -> 0.4196 | Improved |
| increase feedback rounds from 1 to 10 (5 demos) (ablation) | Gemini-2.5-Pro · HeuriGym pickup and delivery | QYI: 0.2350 -> 0.4196 | Improved |
| increase feedback rounds from 5 to 10 (5 demos) (ablation) | Gemini-2.5-Pro · HeuriGym pickup and delivery | QYI: 0.3330 -> 0.4196 | Improved |
| increase iterations allowed from 1 to 10 (ablation) | GPT-o4-mini-high · HeuriGym | solve_III success rate: 53.2% -> 74.8% | Improved |
| increase iterations allowed from 1 to 10 (ablation) | DeepSeek-V3 · HeuriGym | solve_III success rate: 14.2% -> 46.8% | Improved |
| raise sampling temperature from 0 to 1 (ablation) | three representative models · HeuriGym | quality and yield: higher T increases diversity and quality but lowers yield | Mixed |
Closest papers by meaning
- CO-Bench: Benchmarking Language Model Agents in Algorithm Search for Combinatorial Optimization (AAAI 2026 · similarity 0.94)
- PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMs (ICML 2026 · similarity 0.91)
- Large Language Models as End-to-end Combinatorial Optimization Solvers (NeurIPS 2025 · similarity 0.90)
- ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution (NeurIPS 2024 · similarity 0.89)
- Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model (ICML 2024 · similarity 0.88)
- HSEvo: Elevating Automatic Heuristic Design with Diversity-Driven Harmony Search and Genetic Algorithm Using LLMs (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
