Quantify the energy, water, and CO2 of an AI token, including for models whose size is unknown

A methodology measures GPU consumption in inference, separates input and output tokens, and classifies proprietary models into size bands by Monte Carlo sampling.

How much does an inference token consume? The question remains difficult since the most used models are hosted in the cloud and neither their size nor their deployment configuration is public. A methodology published on September 27, 2026 proposes a way to estimate it, distinguishing between input tokens (prefill phase) and output tokens (decoding phase).

The protocol starts from real measurements: the consumption of graphics cards is measured during inference campaigns with open-weight models, on a wide range of text-based tasks. The energy share of the rest of the server, excluding the GPU, is estimated from the execution time. A Bayesian linear regression then relates the energy per token to the size of the model, the request traffic, and the hardware configuration.

The case of closed models

For proprietary boundary models, whose size and deployment are unknown, the authors classify them in size bands based on naming conventions and performance preconceptions, then sample the space of possible configurations using the Monte Carlo method in order to obtain an average energy per token with its uncertainty. Finally, these measurements are used to estimate emissions in CO2 equivalent — both in use and embodied — and water consumed per token, with the idea of providing data on which to act to reduce cost, electricity, emissions and water at the same time.