Technical Terms
Attention Mechanism
Definition
A technique in neural networks that allows the model to focus on the most relevant parts of the input when generating each part of the output. Self-attention, the key innovation in transformers, lets every token attend to every other token.
In Plain English
How AI decides what to pay attention to. When reading "The cat sat on the mat because it was tired," the attention mechanism helps the AI understand that "it" refers to "the cat" — by paying attention to the relationship between those words.