Research map / Architectures and efficiency / RNNs and normalization

Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Dooly profiles LLM inference operations once per signature via taint-propagated single-pass tracing and reuses the serving engine's init code to isolate stateful attention, cutting profiling GPU-hours by 56.4% while predicting vLLM TTFT within 5% and TPOT within 8% MAPE.

Problem

Profile-based LLM inference simulators hardcode their operation sets to a specific software stack and re-profile every operation from scratch for each model and backend, making cross-configuration exploration prohibitively expensive.

Approach

Label each tensor dimension as model-derived or request-derived via taint propagation during one inference pass, use these labels to compute operation signatures so equivalent operations across configurations are profiled once, and isolate stateful attention ops by reusing the serving engine's own initialization code.

Key results

• Dooly predicts vLLM TTFT within 5% MAPE and TPOT within 8% MAPE across reported percentiles on A100.
• Across 12 models with three attention backends, profiling GPU-hours are reduced by 56.4% relative to the existing per-configuration profiling approach.
• Dooly identifies 100 unique signatures versus 612 naive operator signatures across 36 configurations (26.6 GPU-hours naive).

Surprising findings

• Profiling just 4 of 12 models covers over 98% of total profiling time, with later models adding under 0.2 GPU-hours each.
• No single (model, backend) configuration wins across all workloads; winner inversions occur as sequence length grows.

Limitations

• Operations bypassing PyTorch's dispatcher (e.g., Triton kernels) can lack kernel-level taints and rely on module-level coverage.
• Only text prompts are traced; other modalities require adapting entry points.
• MoE kernels are profiled under random expert routing, so data-dependent routing is left to the simulator.

Experimental setup

Datasets: ShareGPT4 synthetic prefill-heavy and decode-heavy workloads

Benchmarks: ShareGPT4 trace simulation synthetic prefill-heavy and decode-heavy per-batch workloads

Metrics: MAPE TTFT error TPOT error profiling GPU-hours scheduling decision agreement

Baselines: Vidur naive per-configuration profiling Revati LLMServingSim

Builds on: Vidur Kineto / PyTorch Profiler vLLM FlashInfer

Closest papers by meaning

• The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.88)
• HCInfer: An Efficient Inference System via Heterogeneous Error Compensation for Resource-Constrained Devices (NeurIPS 2026 · similarity 0.86)
• SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.86)
• Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.86)
• TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.85)
• Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.85)