Every transformer-based AI model since 2017 uses an attention mechanism where compute scales quadratically with context length.

Transformers fixed the amnesia issue by introducing an attention mechanism, allowing the model to look back at any previous word directly and selectively get exactly the information it needed.

The 'attention' mechanism in large language models was first introduced by Google's 'Attention Is All You Need' paper.

Applying attention residuals to top AI models with hundreds of billions or even over a trillion parameters runs into physics limitations due to infrastructure.

The transformer breakthrough occurred in 2017.