Token
The smallest unit of text a language model processes: on average, one token is roughly 3/4 of an English word. It is the billing unit of LLM APIs.
A token is the chunk of text a language model (LLM) actually sees. Before processing any sentence, the model splits it into tokens using a *tokenizer*: common words become a single token, long or rare words split into several pieces, and punctuation and spaces count too. Non-English languages usually produce slightly more tokens per word, because tokenizers are trained mostly on English text.
Tokens matter for two practical reasons:
- Cost: LLM APIs charge per million input tokens (what you send) and output tokens (what the model generates). Estimating tokens means estimating your bill.
- Context limit: each model has a maximum number of tokens it can consider at once (the "context window"). Documents larger than that must be summarized or split.
Concrete example: the phrase "generative artificial intelligence" becomes around 4 to 6 tokens depending on the tokenizer. A 20-page contract can exceed 15,000 tokens, which determines whether it fits in the model's window and how much it costs to process.