The insane engineering of Deepseek V4

Deepseek's team is approximately 40 times smaller than OpenAI's team.

Deepseek V4 Pro has 1.6 trillion parameters, placing it among the best models in the industry.

The 'attention' mechanism in large language models was first introduced by Google's 'Attention Is All You Need' paper.

The key idea behind Deepseek V4's hybrid attention architecture is not to treat all past information as equally important.

Compressed Sparse Attention (CSA) in Deepseek V4 groups small chunks of tokens, such as four at a time, into a single denser representation to reduce sequence length.

Heavily Compressed Attention (HCA) in Deepseek V4 groups approximately 128 tokens or an entire paragraph into a single representation for aggressive compression.

Deepseek V4's sliding window attention keeps the most recent tokens, such as the last 128 words, completely uncompressed with full fidelity.

Deepseek V4 Pro requires 3.7 times lower FLOPs (compute) compared to the previous Deepseek version 3.2.

Deepseek's engineering team introduced a novel architecture called manifold constrained hyperconnections (mHC) to prevent catastrophic spikes in signal when scaling to over a trillion parameters.

Deepseek replaced the industry-standard Adam W optimizer with a custom optimizer called Muon, which uses a two-phase process to accelerate learning and maintain stability.

Deepseek V4's data center choreography breaks down data transfer into smaller sequential waves, overlapping computation and communication to eliminate network latency.

Deepseek V4 Pro is the second-best open-source model, just below Kimik 2.6, according to an independent leaderboard from Artificial Analysis.