Scaling Llama4 Training to 100K - Live from SCC | @Scale

Saif Hasan, a software engineer at Meta, details the challenges and solutions for scaling the training of large language models to 100,000 GPUs. The talk addresses the increasing size of LLMs and the corresponding need for more computational resources, noting that Llama4 requires training on a 100,000 GPU cluster. A significant problem is the increasing initialization time as the job scales, compounded by frequent hardware failures, which can reduce effective training time to 50%. To combat this, Hasan outlines two key strategies: improving collective initialization through optimizations like TCP store-based ring formation and bidirectional AllGathers, and implementing fault-tolerant training. Fault-tolerant training isolates failures to subsets of GPUs, allowing healthy GPUs to continue training, improving effective training time to 90%.

Outlines

Part 1: Scaling Challenges and Inefficiencies

Part 2: Initialization and Performance Optimization

Part 3: Fault-Tolerant Training Strategies

Part 4: Future Outlook and Key Takeaways

Sign in to continue reading, translating and more.

Continue

Scaling Llama4 Training to 100K - Live from SCC

@Scale

Part 1: Scaling Challenges and Inefficiencies

The Evolution and Resource Demands of Large Language Models

Challenges in LLM Training: Failures, Initialization Time, and Effective Training Time

Part 2: Initialization and Performance Optimization

Improving Collective Initialization: Optimizations in Ring Formation

Optimizing AllGathers and Overcoming GPU Resource Limitations with CPU Emulation

CPU Emulation Insights: Identifying Bottlenecks and Optimizing Allgather Implementation

Part 3: Fault-Tolerant Training Strategies

Fault-Tolerant Training: Isolating Failures and Leveraging Data Parallelism

Building a Fault-Tolerant Collective: Coordinator, Quorum Views, and Training Resumption

Part 4: Future Outlook and Key Takeaways

Key Learnings and Future Directions in Scaling LLM Training

Scaling Llama4 Training to 100K - Live from SCC

@Scale

Part 1: Scaling Challenges and Inefficiencies

00:01The Evolution and Resource Demands of Large Language Models

The Evolution and Resource Demands of Large Language Models

00:44Challenges in LLM Training: Failures, Initialization Time, and Effective Training Time

Challenges in LLM Training: Failures, Initialization Time, and Effective Training Time

Part 2: Initialization and Performance Optimization

04:12Improving Collective Initialization: Optimizations in Ring Formation

Improving Collective Initialization: Optimizations in Ring Formation

07:32Optimizing AllGathers and Overcoming GPU Resource Limitations with CPU Emulation

Optimizing AllGathers and Overcoming GPU Resource Limitations with CPU Emulation

10:19CPU Emulation Insights: Identifying Bottlenecks and Optimizing Allgather Implementation

CPU Emulation Insights: Identifying Bottlenecks and Optimizing Allgather Implementation

Part 3: Fault-Tolerant Training Strategies

14:04Fault-Tolerant Training: Isolating Failures and Leveraging Data Parallelism

Fault-Tolerant Training: Isolating Failures and Leveraging Data Parallelism

16:59Building a Fault-Tolerant Collective: Coordinator, Quorum Views, and Training Resumption

Building a Fault-Tolerant Collective: Coordinator, Quorum Views, and Training Resumption

Part 4: Future Outlook and Key Takeaways

19:55Key Learnings and Future Directions in Scaling LLM Training

Key Learnings and Future Directions in Scaling LLM Training