Research map / Architectures and efficiency / RNNs and normalization

Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents Primus Projection, a hybrid benchmark-plus-analytical projection engine for distributed LLM training throughput and memory, and a Primus Tuning Agent that uses it as a scoring oracle with an LLM search loop to find configurations.

Problem

Configuring distributed LLM training across hundreds of GPUs requires searching a huge joint space of parallelism, micro-batch, recomputation, sharding, and overlap settings, and trial-and-error on multi-node clusters wastes GPU-hours on OOMs and suboptimal recipes.

Approach

Benchmark compute kernels on as few as one GPU, downscale-measure-upscale to analytically restore out-of-scope collectives, FSDP overlap, and pipeline schedule at multi-node scale; use this fast projection as an oracle for an LLM-driven cross-axis search with an analytic memory pre-filter.

Key results

• All projections stay within about 10% of measured multi-node throughput; 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% on Qwen3-235B-A22B at 256 GPUs.
• On Mixtral 8x22B the Primus Tuning Agent finds a configuration delivering +27% measured throughput (4,402 vs 3,475 tok/s/GPU) over the AMD 4-node BF16 reference in under 30 minutes of single-node exploration.
• Rank preservation with Spearman rho 0.933 to 1.000 across three MoE models, with 0% selection regret; memory pre-filter rejected 7 of 30 trials before GPU use.

Surprising findings

• Analytical memory pre-filter predictions agreed with hardware: every configuration projected under cap ran, every one projected over cap OOMed.
• Calculon's errors on Llama-3.1-70B were in opposite directions across FP8 and BF16, inflating the precision gap to 2.9x versus 1.5x measured.

Limitations

• Validated only on AMD Instinct GPUs and ROCm; NVIDIA and TPU not empirically validated.
• Network contention is not modeled and could add error in some inter-node EP or sub-node benchmark settings.
• Reported search results show about 10% optimistic bias relative to measured throughput (workload-level calibration offset).

Experimental setup

Benchmarks: AMD ROCm Performance Results (Llama 3.1, Mixtral 8x22B) DeepSeek V3 671B 1024-GPU run Qwen3-235B-A22B 256-GPU run GPT-OSS-120B 64-GPU sweep

Metrics: projection error (%) tok/s/GPU Spearman rho Kendall tau selection regret Avg |Err|

Baselines: Calculon Galvatron-style DP search Claude Code agent AMD published 4-node BF16 reference Bayesian optimization (discussed)

Builds on: Calculon Galvatron ReAct DSPy Recursive Language Model

Closest papers by meaning

• SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.89)
• Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.87)
• ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.87)
• ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.87)
• Heterogeneous Parallelism for Multimodal Large Language Model Training (NeurIPS 2026 · similarity 0.87)
• Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.86)