Tag

profiling.

2 writings found

Latest Archives

Why Your Attention Kernel is 3.7x Slower Than You Think

Understanding PyTorch attention backends through profiler traces reveals why naive implementations beat optimized ones, and what it means for LLM performance.

Profiling Attention in PyTorch: From Naive to FlashAttention

A deep dive into profiling attention mechanisms in PyTorch, comparing naive, SDPA math, efficient, and FlashAttention backends using GPU traces.
View all writings →