Research map / Architectures and efficiency / Neural architecture search
Integrated Hardware Architecture and Device Placement Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Phaze jointly searches accelerator architecture (tensor/vector cores, memory) and distributed device placement for LLM training, using an ILP per-accelerator scheduler and a dynamic program for pipeline/data/tensor parallel placement.
Problem
Hardware accelerator architecture and distributed device placement (data, pipeline, tensor parallelism) for deep learning training have been optimized in isolation, leaving the joint co-optimization unexplored.
Approach
Co-optimizes accelerator architecture and device placement via an ILP that encodes operator finish times as continuous variables (avoiding time-indexed variables), feeding per-layer latencies into a dynamic program for pipeline placement, with an area-based early-stopping architecture search.
Key results
- Phaze-Common architecture and device placement achieves 2.9x higher average throughput than TPUv4 with expert device placement.
- Phaze-Per Model achieves 3.6x higher throughput than TPUv4 with expert device placement on average.
- Phaze-Common delivers 1.8x and 2x higher average throughput against TPUv4 and Spotlight, respectively, when both use Phaze's device placement solver.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Co-optimize per-model accelerator architecture and device placement instead of using TPUv4 with expert placement | Phaze · All evaluated workloads | average throughput: 3.6× higher throughput | Helped |
| Co-optimize a common accelerator architecture and device placement instead of using TPUv4 with expert placement | Phaze · All evaluated workloads | average throughput: 2.9× higher throughput | Helped |
| Use Phaze device placement with a fixed TPUv4 architecture | TPUv4 · Evaluated workloads | throughput | Helped |
| Co-optimize the common architecture and device placement rather than using the baseline configurations | Phaze · All evaluated workloads | average throughput: 1.8× and 2× higher average throughput | Helped |
Surprising findings
- Larger HBM does not always yield higher throughput; Phaze-Common uses 64GB rather than the largest 80GB HBM since throughput is identical or only marginally better.
- Phaze selects architectures with smaller global buffers (4 to 32 MB) than TPUv4's 128 MB, while using more vector cores.
Limitations
- Assumes a homogeneous, flat network topology without hierarchical network modeling.
- Does not overlap cross-operator communication with computation unless operators are fused.
- Does not devise new tensor model parallelism strategies; uses globally fixed parallelism degrees across layers.
Experimental setup
Benchmarks: OPT 350M BertLarge GPT2 1.5B Llama2-7B Megatron 2.5B Megatron 8.3B Megatron GPT3 175B
Metrics: throughput speedup over TPUv4 convergence time
Baselines: TPUv4 Spotlight
Builds on: Piper Alpa WHAM PipeDream-Flush Megatron-LM
Closest papers by meaning
- Device Placement Optimization with Reinforcement Learning (ICML 2017 · similarity 0.88)
- A Hierarchical Model for Device Placement (ICLR 2018 · similarity 0.86)
- An Empirical Study of Distributed Deep Learning Training on Edge (Student Abstract) (AAAI 2024 · similarity 0.85)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.85)
- Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models (AAAI 2024 · similarity 0.84)
- Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.