Posts Tagged "speculative-decoding"

A long-running agent works with 100,000 tokens of context or more, and the GPU memory that holds it is getting scarcer. Our paper, Adaptive DHiSparse, keeps each session's KV cache in host memory and lends it free GPU memory as load allows. On DeepSeek-V4.1-Flash with 500,000-token sessions, the same GPUs produced 1.8 times the output once memory ran out.

We served DeepSeek V4 Pro for two days. What we wanted was hands on the architecture: attention that scales almost linearly instead of quadratically, and a speculative decoder that adapts to load. Both delivered. The preview checkpoint did not.