Research map / Reinforcement learning / Visual and representation RL
LineFlow: A Framework to Learn Active Control of Production Lines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LineFlow, an open-source Python discrete-event simulation framework for production lines with a gymnasium RL interface, and benchmarks PPO, recurrent PPO, TRPO, and A2C against analytical optima on several case studies.
Problem
Active control of production lines (routing, worker reallocation, waiting times) lacks a standardized, general simulation and RL benchmarking framework, and existing studies rely on ad hoc simulators.
Approach
An extensible simulator of production lines built on SimPy with gymnasium-compatible RL interaction at fixed decision intervals, where subproblems are formulated so their optimal policies can be derived analytically and used as reference points for RL agents.
Key results
- On the WT scenario, all tested RL algorithms reach near-optimal reward (e.g., rPPO 157.7 ± 1.3 vs. optimal 156.2 ± 1.5).
- On PD_5, PPO reaches 651.3 ± 107.0 and TRPO 664 ± 77.0 versus an optimal 738.3 ± 3.5, while A2C only reaches 358.3 ± 92.5, showing actor-critic methods fail to learn part distribution.
- On the complex line CL, no algorithm trained from scratch achieved positive reward (agents converged to a deadlock at zero); with curriculum, the best hand-made heuristic reached 254.6 ± 1.1, above the best RL agents.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use recurrent PPO rather than the other tested algorithms | WTJ control · WTJ | reward: 105.9\pm 11.6 vs 93.7\pm 6.9 (PPO) | Helped |
| Use A2C rather than policy-based methods | WA worker assignment · WA | reward | Helped |
| Use policy-based approaches rather than actor-critic methods | PD part distribution · PD | reward | Helped |
| Add curriculum training that gradually increases scrap costs | CL PPO agents · CL | reward | Helped |
| Replace recurrent PPO with stacked PPO using past observations | CL agent · CL | reward: 10 times longer | Hurt |
| Keep worker distribution fixed rather than dynamically reallocating workers (ablation) | CL heuristic · CL | reward | Helped |
| Use look-back l=1 for estimating assembly processing time (ablation) | WTJ online waiting-time agent · WTJ | reward: l=1 | Helped |
Surprising findings
- Agents trained from scratch on the complex line converge to a deadlock that stops production, yielding zero reward, and never escape it.
- Adaptive worker reallocation in the complex line produced oscillatory behavior, so a fixed worker distribution outperformed redistribution strategies.
Limitations
- RL agents do not reach optimum on the more complex industrial-scale CL scenario, which requires curriculum learning and memory.
- Benchmarks are a proof-of-principle with a limited set of algorithms and hyperparameters (three seeds each).
Experimental setup
Benchmarks: WT WTJ WA_{3,9} WA_{4,12} WA_{5,15} PD_3 PD_4 PD_5 CL
Metrics: episode reward (value of produced parts minus scrap cost)
Baselines: PPO TRPO recurrent PPO A2C stacked PPO greedy heuristic
Builds on: PPO SimPy gymnasium stable-baselines3
Closest papers by meaning
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.84)
- Learning to Optimize Permutation Flow Shop Scheduling via Graph-Based Imitation Learning (AAAI 2024 · similarity 0.84)
- LNS2+RL: Combining Multi-agent Reinforcement Learning with Large Neighborhood Search in Multi-agent Path Finding (AAAI 2025 · similarity 0.83)
- Structured Reinforcement Learning for Combinatorial Decision-Making (NeurIPS 2025 · similarity 0.83)
- Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making (ICML 2025 · similarity 0.83)
- Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.