GPU Programming & AI Series

A comprehensive video series covering Triton programming, GPU kernels, and deep learning - with an accompanying textbook. 30 videos planned for 2025.

Roadmap

Planned topics and progress

Part 1: Foundations

  • Introduction to GPU Programming
  • Triton Basics: Your First Kernel
  • Memory Hierarchy & Access Patterns
  • Tiling Strategies

Part 2: Core Operations

  • Matrix Multiplication from Scratch
  • Softmax & Numerical Stability
  • Layer Normalization
  • Convolutions in Triton

Part 3: Attention & Transformers

  • Self-Attention Mechanism
  • Flash Attention: Theory
  • Flash Attention: Implementation
  • Multi-Head Attention

Part 4: Advanced Topics

  • Fused Kernels
  • Custom Autograd Functions
  • Profiling & Optimization
  • Real-World Case Studies

Textbook

Written companion to the video series

The textbook will accompany the video series with deeper mathematical foundations, additional examples, and exercises. Available as the series progresses.

Released Episodes

Videos coming soon. Follow on X for updates.