AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

Andrej Jovanovic, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use a global projection from the aggregated pseudo-gradient without full-rank QHM (ablation)LoRDO-Global · 16M, r=8final perplexity: 37.42 (local) -> 46.81 (global)Hurt
use a global projection without QHM at high rank (ablation)LoRDO-Global · 16M, r=128final perplexity: 35.35 (local) -> 36.65 (global)Hurt
add full-rank quasi-hyperbolic momentum to global projection (ablation)LoRDO-Global · 16M, ranks 8 to 64final perplexityImproved
use low-rank form of QHM instead of full-rank form with global projection (ablation)LoRDO-Global · theory and appendix A.4subspace stagnationNo clear effect
use global projection instead of local projection with QHM (ablation)LoRDO · 16M and 125M, low ranksfinal perplexityImproved
reduce local batch size (more workers, fixed global batch 64) (ablation)LoRDO-Local vs Global · 16M, r=8perplexity gap: Local - Global gap 1.48 to 2.30Hurt
double the global batch size (ablation)LoRDO-Local and Global · 125M, various ranksperplexityImproved
lower synchronization frequency (larger K) (ablation)LoRDO · 16M, r=8 vs r=128final perplexityHurt
add QHM term with higher beta1 under infrequent sync (ablation)LoRDO · 16M, sync frequency sweepfinal perplexityImproved
use LoRDO-Global with infrequent communication vs low-rank DDP720M model, r=256 · 720M pretrainingperplexity: 10.34 -> 10.41No clear effect
restrict to rank 256 instead of full rank 2048DDP at 720M · 720M pretrainingperplexity: 9.85 -> 10.34Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy