The Mathematricks Behind Flash Attention

In this session, we will motivate the need for Flash Attention and discuss its high-level implementation. The focus will be on understanding why traditional attention mechanisms become inefficient at scale and how Flash Attention helps improve performance.

This is a technical session aimed at building intuition around efficient attention mechanisms in modern models.

Key Takeaways:

• Why Flash Attention is needed
• High-level understanding of Flash Attention
• Limitations of standard attention mechanisms
• Practical intuition for efficient attention computation

Datta is a Senior Machine Learning Engineer at Unsloth, working on optimizing LLM fine-tuning for speed and memory efficiency, particularly using techniques like LoRA.

Previously, they worked at Nutanix on enterprise-grade LLM deployment solutions and contributed to open-source projects such as KServe and vLLM. They hold a degree in Mathematics and Computing from IIT Hyderabad and are interested in the intersection of mathematics and machine learning