Send each query to the least carbonized region forgets the cost of recalculating history. By taking this into account, CarbonRouter reduces emissions by an average of 16%.
"Carbon-aware" routers direct each request to an LLM to the region with the cleanest electricity at the time, or to a smaller model. A team of researchers has found a flaw in this reasoning: a conversation is not a series of independent requests. Each round reuses the KV cache of the history, stored only where the previous rounds were served. Moving the session forces you to recalculate this history, an expense that grows with the length of the exchange and can cancel out the gain.
A cache cost in the calculation
Their system, CarbonRouter, chooses the model, GPU type and region for each request by integrating this recalculation cost, based on the number of tokens already cached. A lightweight mechanism also keeps the session in place when it might come back in the next round, which would require everything to be rebuilt a second time.
Tested on two sets of real conversations, with five LLMs, three GPU types, and four regions, it emits 16.1% less CO2 than FairCarbon, the best comparable router, while improving estimated response quality and latency. The saving reaches 23.1% for conversations longer than 30 turns.
Read the full article on the original source
Go to the source →