Language token tax

What it does

It counts the same text in several languages and shows how many times the tokens of the baseline — English, unless you choose otherwise — each language costs. That multiple is the language's token tax: the same meaning, sent to the same model, billed as more tokens.

It starts with Article 1 of the Universal Declaration of Human Rights in 20 languages. You can paste your own parallel texts instead: the same text in each language, with the first one as the baseline.

How it measures

Each text is counted exactly with OpenAI's two encodings, o200k_base (GPT-4o and later) and cl100k_base (the encoding before it), in your browser. The multiplier is one exact count divided by another. The counts are tested against OpenAI's tiktoken, the corpus included.

The chart draws the multipliers for the encoding you choose, largest first, with a line at 1×. The table under it has every figure for both encodings.

Why there is no number for Claude, Gemini or Grok

Their tokenizers do not run outside their providers' APIs. Without them, a count can only be estimated as characters ÷ 4 — the same rule for every language. The ratio of two such estimates is just the ratio of the texts' lengths. It says nothing about how the provider's tokenizer treats a language, so the tool does not show it, and says so instead.

What it does not know

• Other tokenizers. See above.
• Your own text's tax. Article 1 is short, about 33 tokens in English, so one token more or less moves a multiplier by a few hundredths. A longer parallel text of your own gives a steadier figure.
• Translation quality. A longer or shorter translation changes the count. The corpus uses the texts as the UN human rights office's translation project published them.

FAQ

Why is the tax lower with o200k_base than with cl100k_base?

OpenAI's newer encoding has a vocabulary twice as large, with many more pieces of non-English text. For many scripts it needs far fewer tokens for the same words.

Does the tax change what I pay?

Yes, for any provider that bills per token: the same prompt in a language with a 2× tax costs twice as much input, and fills the context window twice as fast.

Why is the text normalised?

Each text is normalised to Unicode NFC, which is canonically the same text. The source's Vietnamese file stores its accents as separate combining marks. A keyboard does not type Vietnamese that way, and that form costs noticeably more tokens.

Sources

The texts are Article 1 of the UDHR, verbatim from the UDHR in XML project at a pinned commit. The project reproduces the translations of the UN human rights office's (OHCHR) UDHR Translation Project; the Thai text is from Thailand's Ministry of Foreign Affairs. No licence is published. The project describes the texts as "part of our commons"; OHCHR's own terms, and those of the Thai source, could not be checked from here and are recorded as unknown. The Declaration itself is UN General Assembly resolution 217 A (III). Each text, its source file and what is known of these terms are Reference Data, listed with their dates in the public tools' data freshness report.