Research map / Multi-agent systems and games / LLM agents
Crucible: Quantifying the Potential of Control Algorithms through LLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Crucible, an LLM-agent framework that simulates multi-level developer tuning of control algorithms and a similarity-weighted metric to quantify their "Tuning Potential."
Problem
Control algorithms are evaluated under default or ideal configurations, ignoring how much their performance can be improved by expert tuning (their Tuning Potential), and there is no systematic way to measure this.
Approach
An LLM agent with varying resource budgets (Bayesian optimization calls and reflection iterations) simulates developers of different skill levels and modifies algorithm logic and parameters; a probe-based environment distance (RMSE over normalized performance vectors) weights performance gains into a formal Tuning Potential score.
Key results
- Crucible's logic-level change lifted Bang-bang on CartPole from 56 to 500 after one LLM modification, a jump not reachable by Bayesian tuning alone.
- Crucible-tuned improvements reach up to 44.1% over Bayesian optimization results on the Puffer dataset in ABR.
- In real-world Dash.js tests, Crucible-tuned HYB and BBA reach QoE 1.72, above the original HYB (1.40), BBA (1.56) and Pensieve (1.66).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply an LLM-driven logic modification to Bang_bang | Bang_bang on CartPole-v1 · CartPole-v1 | score: 56 to 500 | Helped |
| tune HYB with Crucible | HYB · Dash.js real-world WiFi | QoE: 1.40 -> 1.72 | Helped |
| tune BBA with Crucible | BBA · Dash.js real-world WiFi | QoE: 1.56 -> 1.72 | Helped |
| tune Pitree with Crucible | Pitree · Dash.js real-world WiFi | QoE: 1.73 -> 1.73 | No clear effect |
| increase Bayesian optimization iterations from one to two (ablation) | Crucible · ABR | improvement rate: 9.54% to 29.91% | Helped |
| increase Bayesian optimization iterations from one to two (ablation) | Crucible · Scheduling | scenarios failing to achieve performance improvements: 80% to 60% | Helped |
| combine Bayesian optimization with LLM-driven logic changes (ablation) | Crucible · Test scenarios | scenarios with no significant performance gains: 60% to 20% | Helped |
| use Claude 3.7 Sonnet instead of Claude 3.5 Sonnet (ablation) | HYB · ABR (LLM robustness) | performance: 1.03 -> 1.12 | Helped |
| add current bandwidth as a control input and add a bandwidth-based control branch (ablation) | BBA, creating BBA_C · ABR | performance: 4% performance improvement | Helped |
| optimize the simpler SJF algorithm with Crucible instead of Multilevel Feedback (ablation) | Scheduling algorithms · DAG task scheduling | task completion time | Helped |
Surprising findings
- Pure LLM-driven logic changes without Bayesian fine-tuning were unreliable, with about 60% of ABR test scenarios showing no significant gain.
- Decision-tree controller Pitree distilled from RL showed lower tuning potential despite limited initial performance, suggesting logical complexity hinders tuning.
Limitations
- Results depend on the LLM used, so stability varies across models and versions.
- Cannot directly modify internal logic of black-box algorithms; analysis is limited to decision trees distilled from them.
- LLM struggles with fine-grained numerical operations and high-dimensional scheduling inputs.
Experimental setup
Datasets: Oboe FCC 3G Puffer Envivio video trace TPC-H CartPole-v1
Benchmarks: CartPole-v1 ABR simulator with Oboe/FCC/3G/Puffer traces Spark scheduling simulator with TPC-H Dash.js real-world WiFi deployment
Metrics: QoE_lin relative improvement ratio tuning potential P job completion time cart-pole score
Baselines: Bayesian optimization (HYB(B)) BBA BOLA MPC Pitree DQN Pensieve
Builds on: Bayesian optimization Pensieve Pitree
Closest papers by meaning
- Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.84)
- LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling (NeurIPS 2026 · similarity 0.83)
- Parameter-Efficient Fine-Tuning with Controls (ICML 2024 · similarity 0.83)
- The Scaling Laws of Skills in LLM Agent Systems (NeurIPS 2026 · similarity 0.83)
- RF-Agent: Automated Reward Function Design via Language Agent Tree Search (NeurIPS 2025 · similarity 0.83)
- FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.