For years, AI models could not be built very deep because they would be hard to train.

Residual connections allowed AI models to scale from only a few dozen layers to hundreds or even thousands of layers deep.

Models with attention residuals kept improving with increased depth, demonstrating that depth is an advantage, not a limitation.

If AI models are built too deeply, the learning signal flowing backwards through the model would vanish before reaching the beginning, a problem called the vanishing gradient problem.