Research map / Reinforcement learning / Visual and representation RL
EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EarnHFT, a three-stage hierarchical RL framework (Q-teacher-regularized DDQN second-level agents, a diverse agent pool by market trend, and a minute-level DDQN router) for crypto high-frequency trading.
Problem
Applying reinforcement learning to high-frequency trading in crypto markets, where extremely long trajectories (about 1 million steps per horizon) hurt data efficiency and dramatic trend shifts make single agents fail to stay profitable.
Approach
Compute an optimal action-value Q-teacher via dynamic programming on future prices and use a KL term plus an optimal actor to speed up second-level DDQN training; build a pool of trend-specialized agents trained with return-rate-based sampling and select them by profitability; then train a minute-level DDQN router that picks an agent from the pool.
Key results
- EarnHFT achieves the highest total return on all 4 crypto markets, e.g., 19.41% TR on GALA versus 13.95% for the best baseline (IV).
- On BTC/TUSD EarnHFT reaches 0.72% TR and SR 1.22 with 3.07% MDD, while the baselines are negative.
- Ablation on ETH and GALA shows the optimal value supervisor cuts steps to converge to about 15% of DDQN's on GALA and increases return.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EarnHFT instead of the strongest baseline by total return | Crypto HFT · BTCU | TR (%): -2.83 -> 0.72 | Helped |
| use EarnHFT instead of the strongest baseline by total return | Crypto HFT · BTCT | TR (%): 0.47 -> 0.99 | Helped |
| use EarnHFT instead of the strongest baseline by total return | Crypto HFT · ETH | TR (%): -4.29 -> 4.52 | Helped |
| use EarnHFT instead of the strongest baseline by total return | Crypto HFT · GALA | TR (%): 13.95 -> 19.41 | Helped |
| add optimal value supervisor (ablation) | EarnHFT low-level DDQN · GALA | Converged reward sum: -0.01 -> 4.43 | Helped |
| add optimal value supervisor (ablation) | EarnHFT low-level DDQN · ETH | Converged reward sum: -29.6 -> 12.32 | Helped |
| add optimal actor (ablation) | EarnHFT low-level DDQN · GALA | Converged reward sum: 2.89 | Helped |
| add optimal actor (ablation) | EarnHFT low-level DDQN · ETH | Converged reward sum: 4.87 | Helped |
Surprising findings
- Policy-based methods (PPO, DRA) tend to collapse to a dummy policy that holds the current position because of commission fees, performing poorly in bear markets.
- The authors report EarnHFT is a radical trader with higher risk (e.g., 13.89% MDD on ETH), since the optimal supervision only targets profit-related experience.
Limitations
- Risk-related metrics are weaker than profit metrics; the method ignores risk information in its supervision.
- Evaluated only on four crypto markets with a single asset per experiment and a simulated, data-driven trading environment.
- Baselines without public implementations were reimplemented by the authors.
Experimental setup
Datasets: BTC/TUSD second-level crypto data BTC/USDT second-level crypto data ETH/USDT second-level crypto data GALA/USDT second-level crypto data
Benchmarks: BTC/TUSD BTC/USDT ETH/USDT GALA/USDT
Metrics: Total Return Sharpe Ratio Max Drawdown
Baselines: PPO DRA DQN CDQNRP MACD IV
Builds on: DDQN DeepScalper HRPM MetaTrader DTW
Closest papers by meaning
- Logic-Q: Improving Deep Reinforcement Learning-based Quantitative Trading via Program Sketch-based Tuning (AAAI 2025 · similarity 0.86)
- AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning (NeurIPS 2026 · similarity 0.84)
- Trade in Minutes! Rationality-Driven Agentic System for Quantitative Financial Trading (ICLR 2026 · similarity 0.84)
- AlphaQCM: Alpha Discovery in Finance with Distributional Reinforcement Learning (ICML 2025 · similarity 0.84)
- Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets (NeurIPS 2025 · similarity 0.84)
- HyperIV: Real-time Implied Volatility Smoothing (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.