CoolFace
Apppublic

atharv6f/flash-attention-explorer

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face
0likes
App README

FlashAttention Explorer

Interactive demonstrations for the Attention Optimizations article series.

Features

  • —Visualizer: FlashAttention tiling animation and online softmax state tracking
  • —Benchmark: Compare Math, FlashAttention, and Memory-Efficient backends on real models
  • —Prefill vs Decode: Understand why FlashAttention helps prefill more than decode
  • —GQA/MQA: Explore Grouped-Query Attention and KV cache savings
  • —Memory Budget: Plan deployments with memory calculations

Models Used

ModelSizeGQA ConfigUse Case
SmolLM2-360M360M15Q/5KVFast demos, scaling
Qwen2.5-0.5B0.5B14Q/2KVPrefill/Decode
Llama-3.2-1B1B32Q/8KVGQA demonstration

Related Articles

This Space accompanies the "Attention Optimizations" series on HuggingFace:

  1. 1.Standard Attention — The IO Problem
  2. 2.FlashAttention — The Tiling Strategy
  3. 3.FlashAttention — Online Softmax
  4. 4.FlashAttention — IO Analysis

Technical Details

  • —Uses PyTorch SDPA (Scaled Dot Product Attention) with backend switching
  • —Zero GPU for real benchmarks
  • —Gradio Blocks for interactive UI