# Tokens are compression: what Shannon explains about your AI bill

> Why some prompts cost twice as much as others, explained by a gigabyte of ones and a sentence about a fire.

- Author: Marcello Oliveira (https://mrcll.me/about/)
- Published: Nov 8, 2026
- Topics: AI, Fundamentals
- URL: https://mrcll.me/stories/tokens-are-compression/

Every LLM invoice is a count of tokens, in and out. Teams look at it the way they look at a phone bill: a number that goes up. It is more interesting than that. A token count is a measurement of how surprising your text is.

Start with a file where every byte is 1. Make it a gigabyte. How much information does it hold? Almost none. If you know it is all ones, you only need its length. Data and information are different things. Or take the sentence “I saw a fire, and it had flames.” Every fire has flames; the second half can go and nothing is lost. Information is surprise.

A tokenizer works on that idea. Before a model reads anything, the text is compressed into tokens, pieces from a dictionary learned from the training data. Frequent pieces get a token of their own. Rare ones get spelled out in several. So the bill tracks surprise, not length. Try it:

> Interactive figure (TokenMeter): open the web version at https://mrcll.me/stories/tokens-are-compression/

## The arithmetic of “euameiela”

Run the figure on “euameiela”. Nine characters, 72 bits on disk, and only about 21.8 bits of information, 2.419 bits per character. Roughly 70% of the space is wasted. A hand-built prefix code, one bit for the frequent “e”, longer codes for the rare letters, squeezes it to 22 bits. That is as low as it goes: bits are whole numbers and entropy is the floor.

Byte-pair encoding is the same trick at scale. Instead of giving “e” a short code, it gives common chunks of text a single token. Then paste a UUID, a base64 blob or a minified JSON into the box above. Entropy per character climbs, the words stop looking like words, and the token count runs ahead of the character count.

Language matters too. With mostly English training data, an English sentence usually costs fewer tokens than the same sentence in Portuguese. The two presets show it.

What I do with this: I treat IDs, hashes and noisy JSON in prompts as the expensive part they are. I measure with the provider’s own tokenizer in every language we ship. And anything sent on every request gets cached, shortened or removed, because it is a line in the budget whether anyone reads it or not.