Latest Pytorch Podcast Episodes
Batching
PyTorch Developer Podcast - August 18, 2021 13:00 - 13 minutes ★★★★★ - 35 ratingsPyTorch operates on its input data in a batched manner, typically processing multiple batches of an input at once (rather than once at a time, as would be the case in typical programming). In this podcast, we talk a little about the implications of batching operations in this way, and then also ...
Multiple dispatch in __torch_function__
PyTorch Developer Podcast - August 10, 2021 13:00 - 14 minutes ★★★★★ - 35 ratingsPython is a single dispatch OO language, but there are some operations such as binary magic methods which implement a simple form of multiple dispatch. torch_function__ (through its Numpy predecessor __array_function) generalizes this mechanism so that invocations of torch.add with different sub...
Multithreading
PyTorch Developer Podcast - August 03, 2021 13:00 - 18 minutes ★★★★★ - 35 ratingsWriting multithreading code has always been a pain, and in PyTorch there are buckets and buckets of multithreading related issues you have to be aware about and deal with when writing code that makes use of it. We'll cover how you interface with multithreading in PyTorch, what goes into implemen...
Asynchronous versus synchronous execution
PyTorch Developer Podcast - July 27, 2021 13:00 - 15 minutes ★★★★★ - 35 ratingsCUDA is asynchronous, CPU is synchronous. Making them play well together can be one of the more thorny and easy to get wrong aspects of the PyTorch API. I talk about why non_blocking is difficult to use correctly, a hypothetical "asynchronous CPU" device which would help smooth over some of the ...
gradcheck
PyTorch Developer Podcast - July 23, 2021 13:00 - 16 minutes ★★★★★ - 35 ratingsWe talk about gradcheck, the property based testing mechanism that we use to verify the correctness of analytic gradient formulas in PyTorch. I'll talk a bit about testing in general, property based testing and why gradcheck is a particularly useful property based test. There will be some calcul...
torch.use_deterministic_algorithms
PyTorch Developer Podcast - July 21, 2021 13:00 - 10 minutes ★★★★★ - 35 ratingstorch.use_deterministic_algorithms lets you force PyTorch to use deterministic algorithms. It's very useful for debugging! There are some errors in the recording: the feature is called torch.use_deterministic_algorithms, and there is not actually a capability to warn (this was in an old version...
Reference counting
PyTorch Developer Podcast - July 20, 2021 13:00 - 15 minutes ★★★★★ - 35 ratingsReference counting is a common memory management technique in C++ but PyTorch does its reference counting in a slightly idiosyncratic way using intrusive_ptr. We'll talk about why intrusive_ptr exists, the reason why refcount bumps are slow in C++ (but not in Python), what's up with const Tensor...
Memory layout
PyTorch Developer Podcast - July 13, 2021 13:00 - 16 minutes ★★★★★ - 35 ratingsMemory layout specifies how the logical multi-dimensional tensor maps its elements onto physical linear memory. Some layouts admit more efficient implementations, e.g., NCHW versus NHWC. Memory layout makes use of striding to allow users to conveniently represent their tensors with different phy...
pytorch-probot
PyTorch Developer Podcast - July 12, 2021 14:28 - 13 minutes ★★★★★ - 35 ratingspytorch-probot is a GitHub application that we use to automate common tasks in GitHub. I talk about what it does and some design philosophy for it. Repo is at: https://github.com/pytorch/pytorch-probot
API design via lexical and dynamic scoping
PyTorch Developer Podcast - July 09, 2021 13:00 - 21 minutes ★★★★★ - 35 ratingsLexical and dynamic scoping are useful tools to reason about various API design choices in PyTorch, related to context managers, global flags, dynamic dispatch, and how to deal with BC-breaking changes. I'll walk through three case studies, one from Python itself (changing the meaning of divisio...
Intro to distributed
PyTorch Developer Podcast - July 08, 2021 13:00 - 15 minutes ★★★★★ - 35 ratingsToday, Shen Li (mrshenli) joins me to talk about distributed computation in PyTorch. What is distributed? What kinds of things go into making distributed work in PyTorch? What's up with all of the optimizations people want to do here? Further reading. PyTorch distributed overview https://pytor...
Double backwards
PyTorch Developer Podcast - July 07, 2021 13:00 - 16 minutes ★★★★★ - 35 ratingsDouble backwards is PyTorch's way of implementing higher order differentiation. Why might you want it? How does it work? What are some of the weird things that happen when you do this? Further reading. Epic PR that added double backwards support for convolution initially https://github.com/pyt...
Functional modules
PyTorch Developer Podcast - July 06, 2021 13:00 - 14 minutes ★★★★★ - 35 ratingsFunctional modules are a proposed mechanism to take PyTorch's existing NN module API and transform it into a functional form, where all the parameters are explicit argument. Why would you want to do this? What does functorch have to do with it? How come PyTorch's existing APIs don't seem to nee...
CUDA graphs
PyTorch Developer Podcast - June 28, 2021 13:00 - 13 minutes ★★★★★ - 35 ratingsWhat are CUDA graphs? How are they implemented? What does it take to actually use them in PyTorch? Further reading. NVIDIA has docs on CUDA graphs https://developer.nvidia.com/blog/cuda-graphs/ Nuts and bolts implementation PRs from mcarilli: https://github.com/pytorch/pytorch/pull/51436 http...
Default arguments
PyTorch Developer Podcast - June 25, 2021 13:00 - 14 minutes ★★★★★ - 35 ratingsWhat do default arguments have to do with PyTorch design? Why are default arguments great for clients (call sites) but not for servers (implementation sites)? In what sense are default arguments a canonicalization to max arity? What problems does this canonicalization cause? Can you canonicalize...
Anatomy of a domain library
PyTorch Developer Podcast - June 24, 2021 13:00 - 16 minutes ★★★★★ - 35 ratingsWhat's a domain library? Why do they exist? What do they do for you? What should you know about developing in PyTorch main library versus in a domain library? How coupled are they with PyTorch as a whole? What's cool about working on domain libraries? Further reading. The classic trio of domai...
TensorAccessor
PyTorch Developer Podcast - June 23, 2021 13:00 - 11 minutes ★★★★★ - 35 ratingsWhat's TensorAccessor? Why not just use a raw pointer? What's PackedTensorAccessor? What are some future directions for mixing statically typed and typed erase code inside PyTorch proper? Further reading. TensorAccessor source code, short and sweet https://github.com/pytorch/pytorch/blob/mast...
Random number generators
PyTorch Developer Podcast - June 22, 2021 13:00 - 14 minutes ★★★★★ - 35 ratingsWhy are RNGs important? What is the generator concept? How do PyTorch's CPU and CUDA RNGs differ? What are some of the reasons why Philox is a good RNG for CUDA? Why doesn't the generator class have virtual methods for getting random numbers? What's with the next normal double and what does it h...
vmap
PyTorch Developer Podcast - June 21, 2021 13:00 - 17 minutes ★★★★★ - 35 ratingsWhat is vmap? How is it implemented? How does our implementation compare to JAX's? What is a good way of understanding what vmap does? What's up with random numbers? Why are there some issues with the vmap that PyTorch currently ships? Further reading. Tracking issue for vmap support https://g...
Expect tests
PyTorch Developer Podcast - June 18, 2021 13:00 - 13 minutes ★★★★★ - 35 ratingsWhat's an expect test? Why should you use them? Why is inline expect test better than out of line? How to write a good expect test? Further reading. expecttest source implementation https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/expecttest.py (only 311 lines!)
XLA
PyTorch Developer Podcast - June 17, 2021 13:00 - 15 minutes ★★★★★ - 35 ratingsWhat's PyTorch XLA? Why should you care? How is it implemented? How does PyTorch XLA trade off functionality versus ease of performance debugging? What are some new developments in this space? Further reading. XLA's repo has lots of really good docs. Check out https://github.com/pytorch/xla/bl...
TH
PyTorch Developer Podcast - June 16, 2021 13:00 - 11 minutes ★★★★★ - 35 ratingsWhat is TH? Why might you care? What is so horrible about it? What the heck is the generic/ folder? Why are we porting everything to C++? What are some downsides of having ported all our TH code to C++? Further reading. The TH to ATen porting guide has lots of explanations of old school TH id...
TorchScript
PyTorch Developer Podcast - June 15, 2021 13:00 - 19 minutes ★★★★★ - 35 ratingsThere is a really good TorchScript overview at https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/OVERVIEW.md and in this 20min podcast, I want to give you some of the highlights from this document.
CMake
PyTorch Developer Podcast - June 14, 2021 13:00 - 17 minutes ★★★★★ - 35 ratingsWhy is PyTorch's build so g-dang complicated. How to avoid having to deal with cmake at all? And if you have to deal with cmake, what are the most important things to know? And if you were going to improve our cmake, how would you go about doing it... Further reading. The official CMake docume...
torchdeploy
PyTorch Developer Podcast - June 11, 2021 13:00 - 13 minutes ★★★★★ - 35 ratingstorchdeploy is a way of running multiple Python interpreters inside the same process. It can be used to deploy Python PyTorch programs in situations where the GIL is a problem, not the CPython interpreter. How does it work, and what kind of challenges does it pose for people who want to write co...
C++ frontend
PyTorch Developer Podcast - June 10, 2021 13:00 - 17 minutes ★★★★★ - 35 ratingsWhat's the C++ frontend? Why is avoiding templates so important? Why is Tensor a reference type? How do we simulate keyword arguments in C++? Where did the nn Module support in the C++ API come from? Why did we reimplement all modules in C++? How are modules implemented in C++? What are some per...
PyObject preservation
PyTorch Developer Podcast - June 09, 2021 13:00 - 16 minutes ★★★★★ - 35 ratingsGiven two separately refcounted objects, how can you arrange for each of them to stay live so long as the other is live? Why doesn't just having a strong-strong or strong-weak reference between the two objects work? What is object resurrection in CPython? What's a finalizer and why does it make ...
Mobile selective build
PyTorch Developer Podcast - June 08, 2021 13:00 - 16 minutes ★★★★★ - 35 ratingsWhat is mobile selective build? Why are we so obsessed with reducing binary size? How does selective build work? Why doesn't static linking just work? Why can't you just read out the ops used in a TorchScript model to determine what operators you actually need? What are the tradeoffs of statical...
torch.nn
PyTorch Developer Podcast - June 07, 2021 13:00 - 14 minutes ★★★★★ - 35 ratingsWhat goes into the implementation of torch.nn? Why do NN modules exist in the first place? What's the function of Parameter? How do modules actually track all the parameters in question? What is all of the goop in the top level NN module class? What are some new developments in torch.nn modules?...
Code generation
PyTorch Developer Podcast - June 04, 2021 13:00 - 16 minutes ★★★★★ - 35 ratingsWhy does PyTorch use code generation as part of its build process? Why doesn't it use C++ templates? What things is code generation used for? What are the pros/consof using code generation? What are some other ways to do the same things we currently do with code generation? Further reading. To...
Related Pytorch Topics