Research map / Large language models / Reasoning and chain of thought

DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formulates the deterministic FlashAttention-3 backward pass as a DAG critical-path scheduling problem and proposes DASH schedules (descending Q-tile iteration and shift/symmetric shift scheduling) that narrow the throughput gap of deterministic attention on H800 GPUs.

Problem

Deterministic attention backward passes (FlashAttention-3 deterministic mode) lose up to 37.9% throughput because serialized dQ gradient reductions across SMs create pipeline stalls.

Approach

Model the deterministic backward pass as a DAG whose critical path must be minimized, and derive conflict-free scheduling orders (reversed Q-tile traversal for causal masks; cyclic shift and symmetric shift for full/causal masks) that align compute tile order with the fixed reduction order.

Key results

• Proposed schedules improve attention backward throughput by up to 1.28x over the FlashAttention-3 deterministic baseline on NVIDIA H800 GPUs.
• End-to-end transformer block speedups of roughly 2-10% for causal models and about 4% for full-mask models, averaging around 5%.
• Deterministic kernels give bitwise identical gradients (max deviation 0) while non-deterministic kernels deviate by about 2.4e-4 (full) and 4.9e-4 (causal).

Surprising findings

• Shift Scheduling, though theoretically optimal, slightly underperforms the baseline at sequence length 16,384 due to remote L2 cache synchronization latency.
• At headdim=128 the simpler descending Q-tile schedule beats the theoretically optimal symmetric shift schedule, attributed to register spilling.

Limitations

• The DAG model is a simplified abstraction that assumes zero-cost dependency edges and constant phase times, ignoring L2 latency.
• Symmetric shift causal schedule causes register pressure at headdim=128, limiting its practical gains.
• Evaluated only on NVIDIA H800 GPUs with FlashAttention-3 kernels.

Experimental setup

Benchmarks: FlashAttention-3 backward throughput benchmark (H800, BF16 random inputs) end-to-end transformer block runtime

Metrics: throughput (TFLOPS) speedup max gradient deviation

Baselines: FlashAttention-3 deterministic backward Triton tutorial causal attention kernel

Builds on: FlashAttention-3 FlashAttention Triton

Closest papers by meaning

• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.87)
• Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.87)
• A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.87)
• SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference (NeurIPS 2026 · similarity 0.86)
• Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention (ICML 2024 · similarity 0.86)
• ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.86)