Research map / Reinforcement learning / RL from feedback
ERFSL: An Efficient Reward Function Searcher via Large Language Models for Custom-Environment Multi-Objective Reinforcement Learning (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ERFSL, an LLM-based searcher that generates and corrects reward components from user requirements and iteratively tunes their weights for a multi-AUV underwater data collection task trained with TD3.
Problem
Designing reward functions for custom-environment multi-objective RL is hard because LLM-generated reward code can be incorrect and reward component weights can be imbalanced.
Approach
Decomposes LLM-aided reward design into per-requirement reward component generation with component-level testing and a reward critic, plus a weight search that uses training-log feedback and genetic-algorithm-inspired mutation and crossover of weight groups.
Key results
- Reward weight search meets user requirements in about 0.4 iterations on average (RWI) and 1.2 with the numerical reference removed (RWI-UB), per Table 1.
- With a 500x energy weight penalty turned off, GPT-4o needs on average 5.2 weight-search iterations to recover, versus more for EUREKA-M.
- ERFSL with GPT-4o mini still surpasses EUREKA-M in final performance, though the weight initializer does not function with it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove numerical reference and balance requirements from the reward weight initializer prompt (ablation) | ERFSL with GPT-4o · Underwater data collection task | weight-search iterations: 0.40±0.49 -> 1.20±0.75 | Hurt |
| remove the training log analyzer (ablation) | ERFSL with GPT-4o · Underwater data collection task (RWI-UB) | weight-search iterations: 1.20±0.75 -> 1.60±1.02 | Hurt |
| remove the training log analyzer (ablation) | ERFSL with GPT-4o · Underwater data collection task (500x off) | weight-search iterations: 5.20±1.46 -> 6.40±1.86 | Hurt |
| use GPT-4o mini instead of GPT-4o | ERFSL · Underwater data collection task (RWI-UB) | weight-search iterations: 1.20±0.75 -> 1.80±1.17 | Hurt |
| use GPT-4o mini instead of GPT-4o | ERFSL · Underwater data collection task (500x off) | weight-search iterations: 5.20±1.46 -> 8.60±1.74 | Hurt |
| increase the energy weight by a factor of 500 | ERFSL with GPT-4o · Underwater data collection task | weight-search iterations: 5.20±1.46 | Hurt |
| split reward generation into components and give each component explicit feedback | ERFSL reward code generation · Underwater data collection task | feedback iterations per component: one feedback | Helped |
| use ERFSL rather than EUREKA-M for reward weight search | Underwater data collection task · Underwater data collection task | weight-search iterations | Helped |
| remove numerical reference and balance requirements from the reward weight initializer prompt (ablation) | ERFSL · Underwater data collection task | weight-search iterations: 0-3 searches | Helped |
| use GPT-4o mini instead of GPT-4o | ERFSL reward code generation and weight search · Underwater data collection task | final performance | Helped |
| initialize reward weights with the reward weight initializer | ERFSL · Underwater data collection task | Pareto solutions: Two of the five | Helped |
Surprising findings
- The weight initializer does not function with GPT-4o mini, and step-size selection lacks flexibility for that model.
Limitations
- Evaluated on a single underwater multi-AUV data collection task.
- Performance degrades somewhat with smaller LLMs such as GPT-4o mini.
Experimental setup
Datasets: custom underwater multi-AUV data collection environment
Benchmarks: underwater multi-AUV data collection task
Metrics: number of weight search iterations average data overflow energy consumption Pareto solution quality
Baselines: EUREKA-M
Builds on: Eureka TD3
Closest papers by meaning
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.88)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.88)
- Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.88)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.87)
- ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.