Research map / Large language models / Reasoning and chain of thought
Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies the latency-quality trade-off for LLM agents in latency-sensitive decision tasks, introduces HFTBench and StreetFighter benchmarks, and proposes FPX, a layer-wise mixed FP8/FP4 inference method that trades model size and bitwidth for speed.
Problem
LLM agents deployed in real-time, latency-sensitive environments (high-frequency trading, competitive gaming) must balance response speed against decision quality, but this trade-off has been underexplored and existing fixed-size or static quantization options offer only coarse choices.
Approach
Adaptive mixed-precision inference that ranks linear layers by FP4 quantization error (relative to FP16 on calibration data) and assigns FP4 to the most robust fraction gamma of layers while keeping the rest in FP8, combined with choosing model size, giving a fine-grained latency-quality knob.
Key results
- On HFTBench, 14B+FPX (gamma=0.2, 7.2 avg bits) reaches 713 ms latency and 26.52% daily yield versus 23.14% for 14B FP8 at 801 ms.
- On StreetFighter, 3B+FPX (gamma=0.3) reaches 80.0% win rate against FP16 at 200 ms, versus 72.5% for FP8 at 222 ms.
- Pure FP4 for Qwen2.5-14B yields zero daily yield on HFTBench, showing that aggressive quantization without layer selection destroys performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add mixed FP8/FP4 precision with γ=0.2 to Qwen2.5-14B (ablation) | Qwen2.5-14B on HFTBench · HFTBench | daily yield (%): 23.14 -> 26.52 | Helped |
| Increase FP4 compression from γ=0.2 to γ=0.4 (ablation) | Qwen2.5-14B with FPX on HFTBench · HFTBench | daily yield (%): 26.52 -> 12.93 | Hurt |
| Increase FP4 compression to γ=0.6, γ=0.8, or γ=1.0 (ablation) | Qwen2.5-14B with FPX on HFTBench · HFTBench | daily yield (%): 26.52 -> 0.00 | Hurt |
| Add mixed FP8/FP4 precision with γ=0.2 to Qwen2.5-3B (ablation) | Qwen2.5-3B on Street Fighter · Street Fighter | win rate (%): 72.5 -> 77.5 | Helped |
| Increase FP4 compression to γ=0.3 (ablation) | Qwen2.5-3B with FPX on Street Fighter · Street Fighter | win rate (%): 77.5 -> 80.0 | Helped |
| Increase FP4 compression from γ=0.3 to γ=0.4 (ablation) | Qwen2.5-3B with FPX on Street Fighter · Street Fighter | win rate (%): 80.0 -> 62.5 | Hurt |
| Increase FP4 compression to γ=0.6, γ=0.8, or γ=1.0 (ablation) | Qwen2.5-3B with FPX on Street Fighter · Street Fighter | win rate (%): 80.0 -> 12.5; 0.0; 0.0 | Hurt |
| Use FPX at the best reported setting instead of FP8 | Qwen2.5-14B on HFTBench · HFTBench | daily yield (%): 23.14 -> 26.52 | Helped |
| Use FPX at the best reported setting instead of FP8 | Qwen2.5-3B on Street Fighter · Street Fighter | ELO score: 2.19 -> 5.99 | Helped |
| Reduce latency using FPX on Qwen2.5-7B | Qwen2.5-7B on HFTBench · HFTBench | daily yield (%): -12.94 -> -7.25 | Hurt |
Surprising findings
- Further reducing latency of the 7B model on HFTBench harms performance, since faster but weaker decisions increase losses.
- Pure FP4 quantization completely destroys model output quality for the 14B model (zero yield on HFTBench, win rate 0% on StreetFighter).
Limitations
- Precision assignment is at the layer level rather than token level, which may leave trade-offs on the table.
- Evaluation covers only two environments and the Qwen2.5 family, limiting generality across model families.
- Results reported for a single trading period (Aug 5 2024) and a limited number of matches (40 per pair).
Experimental setup
Datasets: Wikitext-2 Polygon.io per-second trading data (Nvidia and Amazon, Aug 5 2024) DIAMBRA Street Fighter environment
Benchmarks: HFTBench StreetFighter
Metrics: daily yield (%) ELO score win rate (%) latency (ms) perplexity
Baselines: FP16 FP8 FP4 W4A16 (int)
Builds on: SVDQuant vLLM SGLang ELO rating
Closest papers by meaning
- LLM-PySC2: Starcraft II learning environment for Large Language Models (NeurIPS 2025 · similarity 0.88)
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.88)
- Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated Negotiation (ICML 2026 · similarity 0.87)
- MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games (ICML 2026 · similarity 0.87)
- Do LLM Agents Have Regret? A Case Study in Online Learning and Games (ICLR 2025 · similarity 0.87)
- Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.