What is Attention mechanism?
The Transformer mechanism in which each token gathers information by computing its relationship to every other token. Its cost grows with the square of sequence length.
Field: AI Hardware & Software Stack
Related terms
AI compiler AI inference All-reduce Arithmetic intensity Backpropagation Batch size Computational graph Data parallelism FlashAttention Gradient checkpointing