og.cc.cd / guides / how-llm-tokens-work

How LLM Tokens Work — and How to Estimate API Cost Before You Ship

By og.cc.cd · Published · Updated · 5 min read

TL;DR: A token is a tokenizer-specific chunk of text, not a word or character; count your real prompt, then multiply input and output tokens by each model’s per-million price to get a defensible budget.

Updated · All sources ↓

Every LLM API bills by the token, but a token is not a word or a character. This guide explains what tokens are, why the same text produces different counts on different models, and how to turn a token count into a monthly budget you can defend.

Open the Token Counter →All guides

What a token actually is

Models do not read letters. Before your prompt reaches the model, a tokenizer splits it into pieces from a fixed vocabulary and replaces each piece with an integer ID. Common English words are usually one piece; rarer words are split into several; spaces and punctuation are often glued to the next word. Chinese, Japanese and code are split differently again.

You can see this on our token counter: type hello world and the o200k_base encoding returns 2 tokens — hello and world (with its leading space). Paste a paragraph of Chinese and the count per character is noticeably higher than for English.

Useful rules of thumb (from the vendors)

Exact counts depend on the tokenizer, but the vendors publish rough ratios you can use for a first estimate:

  • Google: for Gemini models a token is about 4 characters, and 100 tokens is about 60–80 English words (Gemini token docs).
  • DeepSeek: 1 English character ≈ 0.3 token and 1 Chinese character ≈ 0.6 token (DeepSeek token usage).
  • Anthropic: Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text than earlier Claude models (Claude pricing).

Treat these as planning numbers only. A JSON payload, a stack trace or a minified file can tokenize very differently from prose.

Why the same text costs a different number of tokens per model

Each vendor trains its own tokenizer with its own vocabulary. OpenAI’s o200k_base (used by GPT-4o and later OpenAI models) has a large vocabulary that handles many languages efficiently; cl100k_base is the older, smaller one. Claude, Gemini, DeepSeek and Qwen use their own tokenizers. So “1,000 tokens” on one model is not exactly 1,000 tokens on another, and per-token prices are only comparable after you convert to the same text.

Practical consequence: when you compare a Claude 4.7+ model with a GPT model at the same per-token price, the Claude request will usually cost more for the same prompt, because the same text produces roughly 30% more tokens.

How to get an exact count

For planning, a local count is enough. For billing-grade numbers, ask the vendor:

  • OpenAI — POST /v1/responses/input_tokens accepts the same payload as the Responses API and returns the exact input count, including tools, images and the formatting tokens that wrap each message (OpenAI token counting).
  • Anthropic — POST /v1/messages/count_tokens takes a normal Messages request (system prompt, tools, images, PDFs). Anthropic notes the result is an estimate that may differ slightly from the billed number (Claude token counting).
  • Google — models.countTokens returns the input count before you send, and each response’s usage reports input, output, thinking and cached tokens separately (Gemini token counting).
  • DeepSeek — offers a downloadable offline tokenizer, and the usage field of every response is the source of truth.

OpenAI’s own docs point out why local tokenizers like tiktoken fall short for real requests: they do not see images, files, tool schemas or the per-message structure. Count your whole request, not just the user’s text.

Tokens also define your limits

Tokens are not only a billing unit. Every model has a context window (the maximum input plus history it can read in one request) and a maximum output, both measured in tokens. DeepSeek, for example, lists a 1M-token context and up to 384K output tokens for its current models. Rate limits are usually expressed in tokens per minute as well, and OpenAI notes that cached input tokens still count toward those limits.

So a token estimate answers three questions at once: will this fit, how fast can I send it, and what will it cost?

The cost formula

Every vendor in our comparison quotes prices in US dollars per million tokens, with separate input and output rates. The basic formula is:

cost per request = input_tokens / 1,000,000 × input_price
                 + output_tokens / 1,000,000 × output_price

monthly cost     = cost per request × requests per month

Three things make real bills higher than this naive number:

  1. Hidden input. System prompts, tool definitions, retrieved documents and the whole previous conversation are re-sent on every turn of a chat. In a 20-turn conversation the last request may carry 20× the tokens of the first.
  2. Reasoning tokens. “Thinking” models generate internal reasoning before the visible answer. Those are generated tokens and are generally billed at the output rate — check your vendor’s pricing page — so budget for them in the output column.
  3. Tiered and time-based prices. Some models charge more above a prompt-length threshold (for example Claude Haiku 5.5 above 100K tokens, qwen3.7-plus above 256K), DeepSeek has peak and off-peak rates, and some Gemini prices are scheduled to change on a fixed date.

Worked example

Suppose a support bot sends about 2,000 input tokens (system prompt + retrieved FAQ + question) and receives about 500 output tokens per request, 10,000 times a month. That is 20 million input and 5 million output tokens. Using the standard rates checked on 2026-10-18:

Model$ / 1M in$ / 1M outPer requestPer month
Claude Sonnet 5.5$2.00$10.00$0.0090$90.00
gpt-6.1-sol$2.00$10.00$0.0090$90.00
gemini-3.8-flash$0.75$3.75$0.0034$33.75
deepseek-flash (peak)$0.30$1.20$0.0012$12.00
qwen3.8-flash$0.15$0.47$0.0005$5.35

The arithmetic for the first row: 20 × $2 + 5 × $10 = $90. Remember the tokenizer caveat: those 2,000 o200k tokens would be roughly 2,600 tokens on a Claude 4.7+ model, so the realistic Claude figure is closer to $100–$117 a month for the same text. Run your own prompt through the token counter with your expected output size and request count to get the full table for 16 models.

A checklist before you commit to a budget

  • Measure real requests from logs, not the prompt you typed in a playground.
  • Separate the stable prefix (system prompt, tools, documents) from the part that changes — that split decides how much prompt caching can save. See reducing token usage and prompt caching.
  • Cap output with a max-tokens setting and ask for concise formats; on the models we track, output costs roughly 3× to 8× as much as input.
  • Check whether non-urgent jobs can use a batch API: OpenAI and Anthropic both document a 50% discount for batch processing.
  • Re-check prices on the vendor page before you sign off. Our price comparison lists the date each number was checked.

Sources

Vendor features and prices were checked against these official pages on 2026-10-18. Products change; if something here no longer matches, the official page is right.

Related guides