Token

A token is the chunk of text a language model (LLM) actually sees. Before processing any sentence, the model splits it into tokens using a *tokenizer*: common words become a single token, long or rare words split into several pieces, and punctuation and spaces count too. Non-English languages usually produce slightly more tokens per word, because tokenizers are trained mostly on English text.

Tokens matter for two practical reasons:

• Cost: LLM APIs charge per million input tokens (what you send) and output tokens (what the model generates). Estimating tokens means estimating your bill.
• Context limit: each model has a maximum number of tokens it can consider at once (the "context window"). Documents larger than that must be summarized or split.

Concrete example: the phrase "generative artificial intelligence" becomes around 4 to 6 tokens depending on the tokenizer. A 20-page contract can exceed 15,000 tokens, which determines whether it fits in the model's window and how much it costs to process.