What are tokens in an LLM?
Short answer
Tokens are the small pieces of text a large language model reads and writes: whole words, parts of words, punctuation or spaces. In English one token averages about four characters, or three quarters of a word. Tokens are the unit for model limits (the context window) and for pricing, which is quoted per million input and output tokens.
How text becomes tokens
A tokenizer splits text using a fixed vocabulary learned during training. Common words like “the” are one token; rarer words are split, so “tokenization” might become “token” + “ization”. Code, numbers, non-English text and unusual formatting usually need more tokens per word. Each model family has its own tokenizer, so the same text can have different token counts on different models.
Why tokens matter
- Cost: API prices are per token, and output tokens usually cost several times more than input tokens.
- Limits: the context window and maximum output length are measured in tokens.
- Speed: models generate output one token at a time, so long answers take longer.
Estimating
As a rule of thumb, 1,000 tokens is about 750 English words. For an exact count with a specific tokenizer, and the cost at current prices, paste your prompt into the LLM Token Counter.
