Research map / Image generation / Autoregressive and latent generation
LOB-Bench: Benchmarking Generative AI for Finance - an Application to Limit Order Book Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LOB-Bench, a Python benchmark for distributional evaluation of generative limit-order-book (LOBSTER-format) models, and uses it to compare an S5-based autoregressive model (LOBS5) against a parametric baseline, Coletta, and RWKV models.
Problem
Generative models of limit order book message data lack quantitative, distributional evaluation under autoregressive sampling; cross-entropy does not capture error accumulation (model derailment) during open-loop generation, and prior evaluation relied on qualitative stylized facts.
Approach
A general framework that compares real and generated LOB data via aggregator/scoring functions, histogram-based L1 (total variation) and Wasserstein-1 distances, conditional distributions binned by another statistic, error divergence by unroll horizon, a learned discriminator as adversarial score, market-impact response function comparisons, and a downstream MLP mid-price prediction check.
Key results
- LOBS5 achieves the lowest overall L1 and Wasserstein-1 scores among compared models, beating the baseline and Coletta on almost all scores.
- Average L1 response-function distance Delta R is 2.45 for LOBS5 versus 126 for RWKV-6.
- The discriminator reaches ROC AUC of about 0.83 for LOBS5 on GOOG 2023 data, while baseline and Coletta are near 1 (easily discriminated).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LOBS5 instead of RWKV-6 for generating GOOG data | LOB generative models · GOOG impact response functions | mean L1 impact response distance (ΔR; lower is better): 126 -> 2.45 | Helped |
| Add generated data from Coletta to real training data | mid-price trend forecasting MLP · GOOG | prediction precision and recall | Hurt |
| Add generated data from baseline or LOBS5 to real training data | mid-price trend forecasting MLP · longer prediction horizons | F1-score | No clear effect |
| Use half the default Freedman–Diaconis bin size (ablation) | LOBS5 divergence evaluation · evaluated divergence scores | L1 error | Mixed |
| Use double the default Freedman–Diaconis bin size (ablation) | LOBS5 divergence evaluation · evaluated divergence scores | L1 error | Mixed |
| Vary the default Freedman–Diaconis bin size by halving or doubling it (ablation) | LOB-Bench model comparisons · evaluated divergence scores | model ranking | No clear effect |
Surprising findings
- Adding generated data from Coletta markedly reduces MLP mid-price prediction precision and recall, and generated data from the baseline or LOBS5 has no significant effect at longer horizons, suggesting current generators do not yet aid downstream prediction.
- Coletta fails to produce reasonable results on INTC because it was designed for small-tick stocks.
Limitations
- Market impact analysis only considers events affecting the best bid and ask (touch orders).
- Discriminator scoring is a worst-case adversarial measure that can be dominated by a single mismatched score; not robust to noise perturbations.
- Coletta and RWKV models diverge quickly under autoregressive inference, so their impact curves could not be reported.
Experimental setup
Datasets: LOBSTER GOOG 2022 LOBSTER INTC 2022 LOBSTER GOOG and INTC 2023 test data Coletta training data (three days January 2019)
Benchmarks: LOB-Bench (unconditional and conditional scoring functions on GOOG and INTC) LOB market impact response functions MLP mid-price trend forecasting (LOBCAST)
Metrics: L1 distance (total variation) Wasserstein-1 distance discriminator ROC AUC mean absolute difference of response functions (Delta R) F1-score
Baselines: baseline parametric LOB model Coletta RWKV 4 RWKV 6
Builds on: LOBS5 JAX-LOB LOBCAST Coletta
Closest papers by meaning
- MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model (ICLR 2025 · similarity 0.87)
- Market-GAN: Adding Control to Financial Market Data Generation with Semantic Context (AAAI 2024 · similarity 0.87)
- FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
- Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models (NeurIPS 2026 · similarity 0.84)
- FinRpt: Dataset, Evaluation System and LLM-based Multi-agent Framework for Equity Research Report Generation (AAAI 2026 · similarity 0.84)
- Generating Financial Time Series by Matching Random Convolutional Features (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.