Compressed Sparse Attention (CSA) in Deepseek V4 groups small chunks of tokens, such as four at a time, into a single denser representation to reduce sequence length.

The key idea behind Deepseek V4's hybrid attention architecture is not to treat all past information as equally important.

Deepseek V4's sliding window attention keeps the most recent tokens, such as the last 128 words, completely uncompressed with full fidelity.

Deepseek V4's data center choreography breaks down data transfer into smaller sequential waves, overlapping computation and communication to eliminate network latency.

Deepseek V4 Pro requires 3.7 times lower FLOPs (compute) compared to the previous Deepseek version 3.2.

Deepseek V4 Pro has 1.6 trillion parameters, placing it among the best models in the industry.