The attention mechanism is the core innovation behind the Transformer architecture that powers ChatGPT.

How It Works

• Every token in the input looks at every other token to determine relevance
• 'Self-attention' allows the model to understand context and relationships
• Multi-head attention processes multiple relationship types simultaneously
• This is why transformers handle long-range dependencies far better than older RNN models

The 2017 paper 'Attention Is All You Need' by Vaswani et al. introduced this architecture.

Reference:

Wikipedia: Attention Mechanism

https://en.wikipedia.org/wiki/Attention_(machine_learning)