For years, AI models could not be built very deep because they would be hard to train.

The training of AI models never truly stops, with companies constantly training newer, bigger versions that are not yet released to the public.

Residual connections allowed AI models to scale from only a few dozen layers to hundreds or even thousands of layers deep.

Models with attention residuals kept improving with increased depth, demonstrating that depth is an advantage, not a limitation.

If AI models are built too deeply, the learning signal flowing backwards through the model would vanish before reaching the beginning, a problem called the vanishing gradient problem.