Transformers fixed the amnesia issue by introducing an attention mechanism, allowing the model to look back at any previous word directly and selectively get exactly the information it needed.

The key idea behind Deepseek V4's hybrid attention architecture is not to treat all past information as equally important.

The attention residuals design allows the AI to stay perfectly focused on the most important details by selectively choosing which layers' information to use.

The Kimi team's recent breakthrough fixes the amnesia problem of current AI models.

Every transformer-based AI model since 2017 uses an attention mechanism where compute scales quadratically with context length.

The 'attention' mechanism in large language models was first introduced by Google's 'Attention Is All You Need' paper.