Trends in Deep Learning Hardware: Bill Dally (NVIDIA)

Bill Dally's lecture explores hardware advancements driving deep learning, particularly focusing on NVIDIA's contributions and future directions. He highlights that deep learning's current revolution was enabled by hardware, emphasizing algorithms, data, and computing power as key ingredients. Dally introduces Huang's Law, noting the doubling of inference performance every year for the last decade, largely due to smaller number usage, complex instructions, and process technology improvements. He also touches on the importance of software in making hardware useful, referencing NVIDIA's CUDNN. Looking ahead, Dally discusses optimizing number representation through logarithmic numbers, optimal clipping, and scaling granularity, as well as exploring sparsity and improving memory bandwidth with 3D memory. He also shares insights from accelerator projects like Magnet and Magnetic BERT, demonstrating significant efficiency gains. The lecture concludes with a Q&A session, addressing network optimization techniques, energy savings in complex instructions, and the role of software in clipping techniques.

Outlines

Part 1: Introduction, Background

Part 2: LLM Training, Inference, Methods

Part 3: Architecture, Performance, Huang's Law

Part 4: Scaling, Infrastructure, Software

Part 5: Energy, Efficiency, Hardware Optimization

Part 6: Number Systems, Math, Representations

Part 7: Clipping, Scaling, Sparsity

Part 8: Memory, Overhead, Future Outlook

Sign in to continue reading, translating and more.

Open full episode in Podwise

Paul G. Allen School

Part 1: Introduction, Background

00:05Introduction to Bill Dally: Computer Architecture Pioneer and NVIDIA Chief Scientist

Introduction to Bill Dally: Computer Architecture Pioneer and NVIDIA Chief Scientist

00:27Bill Dally's Accomplishments: Awards, Academies, and Aviation

Bill Dally's Accomplishments: Awards, Academies, and Aviation

01:16Deep Learning's Data Distillation and the Grace Hopper Superchip

Deep Learning's Data Distillation and the Grace Hopper Superchip

Part 2: LLM Training, Inference, Methods

02:34Training and Fine-Tuning Large Language Models: From Undergraduate to Graduate School

Training and Fine-Tuning Large Language Models: From Undergraduate to Graduate School

05:37Inference, Retrieval-Augmented Generation, and Grounding in Documents

Inference, Retrieval-Augmented Generation, and Grounding in Documents

07:18Deep Learning's Three Ingredients: Algorithms, Data, and Hardware

Deep Learning's Three Ingredients: Algorithms, Data, and Hardware

08:41The Exponential Growth of Compute Demand in Deep Learning

The Exponential Growth of Compute Demand in Deep Learning

Part 3: Architecture, Performance, Huang's Law

10:06Huang's Law: Doubling Inference Performance Annually Through Smaller Numbers

Huang's Law: Doubling Inference Performance Annually Through Smaller Numbers

11:56Complex Instructions, Process Technology, and Architectural Improvements

Complex Instructions, Process Technology, and Architectural Improvements

13:35Sparsity, Algorithm Efficiency, and Model Optimization

Sparsity, Algorithm Efficiency, and Model Optimization

14:28The Importance of Complex Instructions and Programmable GPUs

The Importance of Complex Instructions and Programmable GPUs

16:51Hopper's Performance, Memory Bandwidth, and Dynamic Programming Instructions

Hopper's Performance, Memory Bandwidth, and Dynamic Programming Instructions

Part 4: Scaling, Infrastructure, Software

18:46Scaling with Multiple GPUs: Model and Data Parallelism

Scaling with Multiple GPUs: Model and Data Parallelism

20:28NVIDIA's Hardware Building Blocks: HGX Servers and DGX Superpods

NVIDIA's Hardware Building Blocks: HGX Servers and DGX Superpods

22:15Bandwidth Provisioning and Network Collectives for Data Parallelism

Bandwidth Provisioning and Network Collectives for Data Parallelism

23:20The Importance of Software in Deep Learning: NVIDIA's AI, HPC, and Graphics Stacks

The Importance of Software in Deep Learning: NVIDIA's AI, HPC, and Graphics Stacks

25:38NVIDIA's Competitive Edge and the AI Hardware Gold Rush

NVIDIA's Competitive Edge and the AI Hardware Gold Rush

Part 5: Energy, Efficiency, Hardware Optimization

26:33Energy Consumption Breakdown in Inference: Math, Memory, and Data Movement

Energy Consumption Breakdown in Inference: Math, Memory, and Data Movement

27:23Improving Number Representation, Sparsity, Tiling, and Memory Efficiency

Improving Number Representation, Sparsity, Tiling, and Memory Efficiency

29:00Communication Circuits and 3D Memory for Increased Bandwidth

Communication Circuits and 3D Memory for Increased Bandwidth

Part 6: Number Systems, Math, Representations

30:03Evaluating Number Systems: Accuracy, Dynamic Range, and Cost

Evaluating Number Systems: Accuracy, Dynamic Range, and Cost

31:11Integer, Floating Point, Logarithmic, and Symbolic Number Representations

Integer, Floating Point, Logarithmic, and Symbolic Number Representations

32:34Spiking and Analog Representations: Drawbacks and Limitations

Spiking and Analog Representations: Drawbacks and Limitations

34:04Representing Weight Distributions: Pruned Networks and Small Values

Representing Weight Distributions: Pruned Networks and Small Values

35:15Logarithmic Representation: Slide Rules and Turning Multiplies into Adds

Logarithmic Representation: Slide Rules and Turning Multiplies into Adds

36:00Logarithmic vs. Floating Point: Accuracy and Dynamic Range

Logarithmic vs. Floating Point: Accuracy and Dynamic Range

37:21Properties of Logarithmic Number Systems: Cheap Multiplies, Hard Adds

Properties of Logarithmic Number Systems: Cheap Multiplies, Hard Adds

39:36Inexpensive Adds in Deep Learning: Factoring Out Table Lookups

Inexpensive Adds in Deep Learning: Factoring Out Table Lookups

Part 7: Clipping, Scaling, Sparsity

41:24Optimally Clipping Numbers: Balancing Clipping and Quantization Noise

Optimally Clipping Numbers: Balancing Clipping and Quantization Noise

43:15Minimizing Mean Squared Error: Iterative Equations and Clipping Factors

Minimizing Mean Squared Error: Iterative Equations and Clipping Factors

45:09The Importance of Scaling: From Layer-by-Layer to Vector-Level Scaling

The Importance of Scaling: From Layer-by-Layer to Vector-Level Scaling

46:15Vector-Level Scaling: Tighter Distributions and Better Results

Vector-Level Scaling: Tighter Distributions and Better Results

48:18Sparsity in Neural Networks: Pruning Weights and Challenges in Implementation

Sparsity in Neural Networks: Pruning Weights and Challenges in Implementation

50:54Structured Sparsity: Regular Patterns and Predictable Computations

Structured Sparsity: Regular Patterns and Predictable Computations

Part 8: Memory, Overhead, Future Outlook

52:55Accelerators vs. GPUs: Data Types, Parallelism, and Memory Optimization

Accelerators vs. GPUs: Data Types, Parallelism, and Memory Optimization