A new preprint models the graphics-processor energy needed to generate language-model output. It separates model weights shared between requests from each request’s growing history and the time taken to produce useful text. Measurements concern processor boards during decoding; model loading, initial input processing and the rest of the facility fall outside that boundary.
Researchers have proposed a method for measuring the energy used while a language model generates text. Their Token-Energy Law preprint distinguishes the work of reading weights shared by several requests from the growing conversation history kept separately for each request. The time taken to produce useful output is another input to the calculation.[1]
The experiments cover dense language models on NVIDIA graphics processors. Measurement starts after the model has loaded and the initial input has been processed. When several boards serve one copy of the model, their energy use is added together while output is counted once. Electricity used while the boards are idle remains in the total. Host processors, networking and the building’s other energy demands are outside the calculation.[1]
The authors calibrate coefficients for each combination of model, hardware, runtime and power policy. They separate calibration conditions from validation conditions and also test what happens as requests finish and leave a batch. Reported agreement with measurements uses the duration actually observed. Estimating energy before a run would additionally require a validated prediction of that duration. The work does not extend its dense-model coefficients directly to models that route requests among different experts.[1]
