AI Blog

by Michele Laurelli

Back to Glossary

Attention Mechanism

/əˈtɛnʃən ˈmɛkənɪzəm/

Technique

Definition

A technique allowing models to focus on specific parts of the input when producing output.

Attention assigns importance weights to different input elements. Used in machine translation, image captioning, and transformers. Enables models to handle long sequences effectively.

Examples

Focusing on relevant words in translation

Highlighting important image regions

Multi-head attention in transformers

Related Terms

Transformer

A neural network architecture based entirely on attention mechanisms, without recurrent or convolutional layers.

Encoder-Decoder

An architecture where an encoder processes input into a representation and a decoder generates output from it.

Michele Laurelli - AI Research & Engineering