Research map / Reinforcement learning / Offline RL
Data Center Cooling System Optimization Using Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a physics-informed offline RL framework with a T-symmetry-enforced GNN thermal dynamics model and latent-space TD3+BC-style policy learning, deployed for closed-loop control of ACUs in a production data center (14-21% cooling energy savings) and compared on a small testbed.
Problem
Reducing data center cooling energy while meeting strict thermal safety constraints, when only limited, narrow-coverage historical operational data and no reliable simulator are available.
Approach
A graph-structured thermal dynamics model (TTDM) with latent forward/reverse ODE dynamics enforcing time-reversal (T-symmetry) consistency provides compact state-action representations; the Q-function and policy are learned in this latent space with TD3+BC-style regularization plus a T-symmetry loss on policy-induced samples.
Key results
- In the production data center, the method reduced ACLF by 14-21% relative to PID across two server rooms without safety violations, over 2000 hours of experiments.
- On the testbed under low and high loads, the method achieved the highest energy efficiency with no CAT violations, while CCA and CQL saved more energy but violated thermal safety.
- Ablations show GNN blocks and T-symmetry enforcement reduce multi-step dynamics prediction MSE and T-symmetry improves policy energy efficiency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed offline RL controller instead of PID | Commercial data center cooling · Server Room A | ACLF (%): 4.42 -> 3.62 (↓18%); 4.32 -> 3.68 (↓15%) | Helped |
| Use the proposed offline RL controller instead of PID | Commercial data center cooling · Server Room B | ACLF (%): 6.03 -> 5.16 (↓14%); 6.04 -> 5.18 (↓14%) | Helped |
| Control all ACUs instead of using PID | Commercial data center cooling · Server Room A | ACLF (%): 3.40 (↓21%) | Helped |
| Control all ACUs instead of using PID | Commercial data center cooling · Server Room B | ACLF (%): 4.85 (↓20%) | Helped |
| Add GNN blocks that encode spatial and control dependencies (ablation) | TTDM · Server Room B historical data | multi-step prediction error (MSE) | Helped |
| Add T-symmetry enforcement to the thermal dynamics model (ablation) | TTDM · Server Room B historical data | multi-step prediction error (MSE) | Helped |
| Add T-symmetry enforcement to offline policy learning (ablation) | Proposed offline RL framework · Testbed, low and high server load | energy efficiency | Helped |
| Multiply the CAT violation penalty weight β₂ by 5 (ablation) | Proposed offline RL framework · Testbed, low server load | ACLF (%): 29.66 -> 29.37 | No clear effect |
| Multiply the CAT violation penalty weight β₂ by 10 (ablation) | Proposed offline RL framework · Testbed, low server load | ACLF (%): 29.66 -> 30.95 | No clear effect |
| Multiply the CAT violation penalty weight β₂ by 5 (ablation) | Proposed offline RL framework · Testbed, high server load | ACLF (%): 26.89 -> 27.50 | No clear effect |
| Multiply the CAT violation penalty weight β₂ by 10 (ablation) | Proposed offline RL framework · Testbed, high server load | ACLF (%): 26.89 -> 26.05 | No clear effect |
Surprising findings
- Aggressive baselines CCA and CQL achieved lower energy consumption on the testbed but violated thermal safety constraints.
- Fan speeds under the proposed policy were lowered while valve openings were slightly increased, reducing ACU energy while maintaining cooling.
Limitations
- Production experiments were restricted to controlling a subset of ACUs and could not run full baseline comparisons for safety reasons.
- Baseline comparisons were limited to a small testbed with lower server load and a stricter 22°C CAT threshold rather than the production environment.
- Only the scale parameter alpha was tuned (range [2.5, 10]), and its effect on conservatism was described qualitatively.
Experimental setup
Datasets: historical operational data from production data center server rooms A and B testbed historical operational data
Benchmarks: production DC ACLF comparison vs PID real-world testbed comparison (ACLF and CAT violation rate)
Metrics: ACLF (air-side cooling load factor) CAT violation proportion multi-step MSE of dynamics model server energy consumption
Baselines: PID MPC (Google data-driven) CCA IQL CQL FISOR
Builds on: TSRL TD3+BC CCA FISOR
Closest papers by meaning
- LC-Opt: Benchmarking Reinforcement Learning and Agentic AI for End-to-End Liquid Cooling Optimization in Data Centers (NeurIPS 2025 · similarity 0.89)
- Carbon Footprint Reduction for Sustainable Data Centers in Real-Time (AAAI 2024 · similarity 0.88)
- Hierarchical Multi-Agent Framework for Carbon-Efficient Liquid-Cooled Data Center Clusters (AAAI 2025 · similarity 0.88)
- Sustainability of Data Center Digital Twins with Reinforcement Learning (AAAI 2024 · similarity 0.87)
- SustainDC: Benchmarking for Sustainable Data Center Control (NeurIPS 2024 · similarity 0.87)
- DCcluster-Opt: Benchmarking Dynamic Multi-Objective Optimization for Geo-Distributed Data Center Workloads (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.