PyTorch Developer Podcast

Compiler collectives

TORCH_TRACE and tlparse

Higher order operators

Inductor - Post-grad FX passes

CUDA graph trees

Min-cut partitioner

AOTInductor

Tensor subclasses and PT2

Compiled autograd

PT2 extension points

Inductor - Define-by-run IR

Unsigned integers

This podcast episode centers around the implementation of unsigned integer support in PyTorch, focusing on the motivations and challenges encountered during the process. The discussion covers technical aspects such as defining semantics for signed and unsigned integer operations, handling integer tensor promotion and o...

Inductor - IR

This podcast episode provides an in-depth look at InductorIR, a tool for representing and compiling PyTorch programs. It delves into the design considerations, challenges, and capabilities of InductorIR, highlighting its role in modeling PyTorch semantics, handling mutable operations, and optimizing performance. The ep...

Dynamo - VariableTracker

This podcast episode discusses the development and optimization of Dynamo, a tool for tracing and optimizing PyTorch programs, through the lens of its variable tracker implementation. Notable changes to the variable tracker include eliminating the need for guard propagation, introducing Mutable Variable Trackers for im...

Unbacked SymInts

This podcast episode explores unbacked cements in PyTorch 2, symbolic integers and conditional branches, unbacked cements for export and non-zero compilation cases, unbacked simmants and broadcasting, broadcasting and shape propagation in TensorFlow, optimization of tensor handling for export, zero-one specialization i...

Zero-one specialization

This podcast episode explores the benefits, challenges, and applications of zero-one specialization in PyTorch, a technique that significantly reduces trace time and simplifies reasoning by specializing certain conditions to be either zero or one, thereby enhancing the performance of neural network models.

torchdynamo

This podcast episode provides an in-depth overview of Dynamo, a Python-based graph representation system that supports tracing and debugging of deep learning models in PyTorch. It compares Dynamo with other tools such as TorchScript and FX symbolic trace, highlighting their strengths and weaknesses. The episode also di...

PyTorch 2.0

History of functorch

Learning rate schedulers

This podcast episode discusses learning rate and learning rate schedulers in PyTorch, covering topics such as their importance, how to choose an appropriate learning rate, the evolution of learning rate scheduler API, the challenges of using stateful APIs, and the advantages of PyTorch's imperative, mutable API design.