The "Tokenizer Tax": Writing to an AI in a language other than English can cost up to 17 times more

A study by the Coalition for Language Equity in AI Research measures the token consumption gap by language: Burmese, Tibetan and Khmer are the most penalized.

Language models charge the token, and not all texts cut out the same way. A study of the Coalition for Language Equity in AI Research, reported by Rest of World, figures this inequality: among the 20 languages most used online, the cost in tokens of the same content varies from 1.5 to 5 times between the most efficient and the most expensive language. Extending to 200 languages, the gap climbs up to 17 times on some models.

Writing counts more than the number of speakers

The writing system weighs more than the size of the population: Burmese, Tibetan, Tifinagh, Khmer, Lao and Chiki are the most disadvantaged. The bill is not alone in question: a more greedy text fills the context window faster, which degrades the quality of responses on long conversations and lengthens latency where the connection is already limited. A young Bangalore shoot thus pays an invisible surcharge on each request, unlike its competitor from Austin.

The authors call for linguistic efficiency to become a product requirement and for large laboratories to publish language-by-language dashboards, with commitments to improve their future tokenizers.