A token is the basic unit of text that large language models process, typically representing a word, part of a word, or a punctuation mark, depending on how the specific model breaks down language. Understanding tokens matters mainly for developers and businesses building custom AI applications, since most AI services price usage based on the number of tokens processed, and models have a maximum number of tokens they can handle at once.
How tokens work
When text is sent to a large language model, it is first broken down, or tokenised, into these smaller units before processing. As a rough guide, one token is roughly three-quarters of an English word on average, though this varies. Every model has a maximum context window, the total number of tokens it can process in a single request including both the input and the generated output, which limits how much text can be analysed or generated at once.
Tokens in practice
- A developer building a custom application on Azure OpenAI monitors token usage closely, since costs scale directly with the volume of text processed.
- A technical team selects a language model with a larger context window specifically because their use case involves analysing very long documents that would exceed a smaller model's token limit.
- A business estimates the running cost of a planned custom AI application by calculating expected token volume based on typical document and query length.
- A development team optimises prompts to use fewer tokens where possible, reducing both cost and response time for a high-volume AI application.
How Advantage factors in token-level considerations
For most UK SMEs using Microsoft Copilot, token-level detail is not something businesses need to manage directly. Advantage factors in token usage and context window limitations when scoping custom AI solutions built on Azure OpenAI or similar platforms, ensuring cost and capability expectations are realistic from the outset.
Frequently Asked Questions
Is a token the same as a word?
Not exactly. A token is often a word, but can also be part of a word, a single character, or punctuation, depending on how the language model breaks down text. As a rough estimate, one token is approximately three-quarters of an English word on average.
Why do tokens matter for business use of AI?
Tokens matter because most AI services, including Azure OpenAI, charge based on the number of tokens processed, and language models have a maximum number of tokens they can handle in a single request, known as the context window. Both of these factors affect cost and what is technically possible in a custom AI application.
Does understanding tokens matter for using Microsoft Copilot day to day?
Not really. Tokens are mostly a backend technical detail for developers building custom AI applications or managing API costs. Everyday users of Microsoft Copilot do not need to think about tokens, since Microsoft handles this complexity behind the scenes.
