Posts Tagged "serving"

A long-running agent works with 100,000 tokens of context or more, and the GPU memory that holds it is getting scarcer. Our paper, Adaptive DHiSparse, keeps each session's KV cache in host memory and lends it free GPU memory as load allows. On DeepSeek-V4.1-Flash with 500,000-token sessions, the same GPUs produced 1.8 times the output once memory ran out.

Last October, ten billion lifetime tokens earned developers a public tribute at OpenAI DevDay. This June, one Umans AI user ran nearly eleven billion tokens in a single day. What that says about token demand, compute supply, and the serving efficiency that has to close the gap between them.

Put the exact same model on GPUs and its token production cost can vary by multiples depending on how you serve it. A walk through the economics of a token: the frontier, the dollars it turns into, the levers that move it, and the claim our measurements let us make.