Shared weights, private history
The same GPU and model can spend more energy per output as conversations grow longer. The Token-Energy Law preprint separates two kinds of memory work behind that change. Model weights are shared by requests decoded together. Key-value state, which represents a request’s preceding text, belongs to that request. Larger batches spread the shared-weight work across more outputs, while longer histories increase private memory work for every output. The service operator is adjusting two different levers.[1]
Time is the third input. When useful outputs take longer, the boards carry the workload for longer. The model puts observed duration alongside shared weights and private history. A processor’s power rating or memory capacity alone therefore cannot settle its energy cost per output. How much work is shared and how quickly that work finishes change together. The denominator is the useful output actually produced, rather than the number of boards installed.[1]
From boards to the facility
The physical boundary is explicit: the measurement covers GPU boards. Model loading and input prefill are excluded, while idle energy is retained. When two boards serve a replica, their energy is added and their shared output is counted once. Host processors, network equipment and facility cooling lie outside that boundary. An operator using the result to budget an electricity bill must add those components separately.[1]
Tests of Qwen and Mistral on Hopper and Blackwell use the same equation structure, but each model and execution configuration receives its own coefficients. Decode duration comes from the experiment as well. That narrows what a buyer can conclude: the preprint does not promise one set of coefficients for every new device and service. A change in execution policy, precision or work distribution changes the configuration that needs measurement.[1]
The service energy calculation
For operations, the implication is to compare serving arrangements alongside hardware. Combining more requests can amortize shared memory work, while the customer’s latency requirement constrains that grouping. Retaining longer histories also increases memory work. Replacing a board with the application’s history requirement unchanged and changing the serving software on an existing board are different interventions. A common workload is needed to determine which intervention delivers the energy saving.[1]
Even if joules per output fall, total consumption still depends on how much output the service produces. Efficiency reduces electricity demand at a fixed volume; growing usage can absorb part or all of that saving. Traffic may instead remain stable and preserve the reduction. The board-level model gives an operator a useful accounting unit. Facility capacity planning must complete it with actual traffic and the loads outside the boards.[1]