Why the Same Text Costs More on GPT Than on Gemini: How AI Tokenizers Actually Work

If you've ever compared API pricing across OpenAI, Anthropic, and Google, you've probably noticed something odd. The same prompt doesn't cost the same amount across providers, even when the per-token price looks similar. The reason isn't just pricing. It's that the same text produces a different number of tokens depending on which model processes it.

This isn't a rounding error. A 1,000-word document might tokenize into 1,200 tokens on GPT-4o, 1,350 tokens on Claude, and 1,100 tokens on Gemini. Over thousands of API calls, that gap adds up fast. Understanding why this happens starts with understanding what a tokenizer actually does.

What a Tokenizer Does

Before a language model can process your text, it needs to break it into smaller pieces called tokens. A token might be a whole word, part of a word, a single character, or even a space. The word "infrastructure" is typically one token. The word "tokenization" gets split into "token" and "ization" by most tokenizers. Numbers, punctuation, and whitespace all get their own treatment.

Each model family ships its own tokenizer with its own vocabulary, a fixed dictionary of every token the model recognizes. When your text arrives, the tokenizer matches it against this dictionary using algorithms like Byte Pair Encoding (BPE). The vocabulary size and the specific merge rules determine how the text gets split.

This is why the same sentence produces different token counts across models. Different dictionary, different splits, different count.

The Vocabulary Size Gap

The single biggest factor in token efficiency is vocabulary size. A larger vocabulary means the tokenizer can represent more words and subword combinations as single tokens, which means fewer tokens for the same text.

Here's where the major model families stand today:

OpenAI's GPT-4o and GPT-5 use the o200k_base tokenizer with roughly 200,000 tokens in its vocabulary. This is OpenAI's newest and most efficient tokenizer. GPT-4 and GPT-3.5, by contrast, still use cl100k_base with about 100,000 tokens. That difference alone means GPT-4o typically produces 5 to 15 percent fewer tokens than GPT-4 for the same input.

Meta's Llama 3 uses a custom tokenizer with a 128,000-token vocabulary, a major upgrade from Llama 2's 32,000 tokens. Alibaba's Qwen 2.5 has a vocabulary of roughly 152,000 tokens. Mistral and DeepSeek V3 each have their own tokenizers in the 32,000 to 128,000 range.

Anthropic and Google don't publish their tokenizer specifications. Based on empirical testing, Claude typically produces roughly 10 to 15 percent more tokens than GPT-4o's o200k_base for the same English text. Gemini tends to be closer, around 5 percent more.

Why This Matters for Your API Bill

API pricing is per token. If Model A tokenizes your prompt into 1,000 tokens and Model B tokenizes the same prompt into 1,150 tokens, Model B is processing 15 percent more tokens, even if both charge the same per-token rate.

Let's make this concrete. Say you're building a RAG pipeline that processes 10,000 documents per day, each averaging 500 words. On GPT-4o at $2.50 per million input tokens, your daily input cost depends entirely on how efficiently the tokenizer handles your content. If Llama 3's tokenizer produces 8 percent more tokens for the same documents, that's 8 percent more compute and 8 percent higher cost at equivalent per-token pricing.

This is especially relevant for multilingual workloads. English text is well-represented in most tokenizer vocabularies, so the differences tend to be moderate. But for languages like Japanese, Korean, Arabic, or Hindi, the gap can be dramatic. A tokenizer trained primarily on English might split a single Chinese character into three or four tokens, while a tokenizer with better multilingual coverage handles it in one.

Subword Tokenization and the BPE Algorithm

Most modern tokenizers use some variant of Byte Pair Encoding. The process starts by treating every character as its own token, then iteratively merging the most frequently co-occurring pairs into new tokens. After enough merges, common words become single tokens, while rare words get split into recognizable subword pieces.

The training data matters enormously here. A tokenizer trained on mostly English code and text will handle Python variable names efficiently but might struggle with medical terminology in German. The vocabulary is frozen after training, so whatever biases exist in the training corpus get baked into every future tokenization.

This is also why newer tokenizers tend to be more efficient. OpenAI's jump from cl100k_base to o200k_base wasn't just about doubling the vocabulary. The training data for o200k_base was broader and more carefully curated, which means better coverage of code, multilingual text, and specialized domains.

Practical Takeaways

If you're evaluating models for production use, don't just compare per-token pricing. Run your actual data through each tokenizer and compare the total token count. The model with the lowest per-token rate might not be the cheapest once you account for tokenizer efficiency.

I built an AI Token Calculator↗ that lets you paste your actual text and compare token counts across nine model families side by side, including GPT-4o, Claude, Gemini, Llama 3, Qwen, Mistral, and DeepSeek. It runs real tokenizers in your browser using WebAssembly, so nothing gets sent to a server. If you're trying to estimate API costs or understand why your token counts don't match across providers, it's worth running your specific workload through it.

The bottom line: tokenization is the hidden variable in every LLM cost calculation. Two models might quote the same price per million tokens, but if one produces 15 percent fewer tokens for your data, that's the one saving you money.

---

Eliran Barhum is an AI infrastructure consultant. Views are his own.