AI Blog

AI Blog

by Michele Laurelli

Back to Glossary

Causal Masking

Technique
Definition

Masking technique preventing attention to future positions in autoregressive models.

Ensures token at position i can only attend to positions ≤ i. Essential for GPT-style models. Implemented as triangular mask.

Examples

1

GPT autoregressive generation

2

Decoder self-attention

3

Preventing future leakage

Related Terms

Transformer

A neural network architecture based entirely on attention mechanisms, without recurrent or convolutional layers.

GPT (Generative Pre-trained Transformer)

A family of large language models developed by OpenAI that use transformer architecture for text generation.

Michele Laurelli - AI Research & Engineering