Multi-Head attention

A mechanism in BERT’s architecture.

This is $O(N^2)$ in software complexity (runtime and memory), where $N$ is the sequence length.

As a result of this, faster solutions and implementations have come, with the most popular one being Flash Attention.