Blog Archive

A long-running agent works with 100,000 tokens of context or more, and the GPU memory that holds it is getting scarcer. Our paper, Adaptive DHiSparse, keeps each session's KV cache in host memory and lends it free GPU memory as load allows. On DeepSeek-V4.1-Flash with 500,000-token sessions, the same GPUs produced 1.8 times the output once memory ran out.

Between late July and late September, a new open-weight model reached our users about every five days. Flash models caught the flagships, memory became the spec that matters, and coding models learned to see. What we shipped, what we turned down, and why.

Last October, ten billion lifetime tokens earned developers a public tribute at OpenAI DevDay. This June, one Umans AI user ran nearly eleven billion tokens in a single day. What that says about token demand, compute supply, and the serving efficiency that has to close the gap between them.

Put the exact same model on GPUs and its token production cost can vary by multiples depending on how you serve it. A walk through the economics of a token: the frontier, the dollars it turns into, the levers that move it, and the claim our measurements let us make.

We served DeepSeek V4 Pro for two days. What we wanted was hands on the architecture: attention that scales almost linearly instead of quadratically, and a speculative decoder that adapts to load. Both delivered. The preview checkpoint did not.

We put an NVFP4 build of GLM-5.2 in front of real users for four days, at 200+ tokens per second. Users called it intoxicating. We retired it anyway: if the tokens are not useful, they are not worth serving, no matter how efficiently we can produce them.

The concrete workflow we use to go from a vague idea to code running in production, with AI agents doing the heavy lifting. Three moves, one increment file, everything in git.

When we launched code.umans.ai at the beginning of February, my cofounder Naji and I saw it as an opportunity to challenge ourselves. We had unlimited LLM access now. No more token anxiety. So we asked: what happens if we really let our agents work autonomously?