Token
A token is the small unit of text, often a word or part of a word, that a language model reads and writes, and it is the unit AI providers use to measure input length, output length and usage costs.
Models do not see letters or whole words. Before any text reaches a large language model, a tokeniser splits it into tokens from a fixed vocabulary. Common words are usually one token; long or rare words are split into several pieces.
In AI, a token has nothing to do with crypto tokens or login tokens. It is purely a measuring unit for text, and a rough rule for English is that one token is about three quarters of a word. Many other languages, including Albanian, often need more tokens for the same meaning.
How it works
The tokeniser turns your text into a list of numbers, one per token. The model processes those numbers and predicts the next token, then the next, building its reply piece by piece. Everything has a token count: your question, any documents you attach, the hidden system prompt and the answer.
The total must fit inside the model's context window. When developers use AI through an API, providers usually charge per token, with output tokens typically costing more than input tokens. You can check token counts yourself. Some AI providers publish a free tokeniser page where you paste text and see it split into pieces with a total, and their APIs report the exact token count of every request. If you use an AI feature through a developer's integration, the provider's usage dashboard shows tokens in and out per day, which is the most reliable way to spot a feature that is sending far more text than it needs.
Tokenisers are built by finding the character combinations that appear most often in the model's training text and giving each one an entry in the vocabulary. English dominated most training sets, so English words split efficiently, while words in many other languages, including Albanian, tend to break into more pieces. Numbers are often split digit by digit or in small chunks, which is one reason models can stumble on arithmetic and long figures.
Example
A clinic builds a small tool that summarises patient feedback forms. Each form is around 300 words, so roughly 400 tokens go in, and a 60-word summary is about 80 tokens out. Multiplying by the number of forms per month gives a clear estimate of what the tool will cost to run.
If the forms arrive in two languages, the clinic can paste a typical one of each into the provider's token counter and compare. The same feedback written in a less common language may need more tokens, and that difference should be part of the cost estimate before the tool goes live. Trimming the forms to the free-text answers, and dropping the repeated header on every page, cuts the input by about a third before any code changes.
Why it matters for your business
Tokens explain three practical things: why long documents get cut off, why API bills scale with how much text you send, and why the same prompt can cost more in some languages than others. If you commission an AI feature, ask for an estimate of tokens per task, not just a monthly fee. For consumer chat plans the same idea shows up as usage limits that run out sooner when you attach long files.
Common mistakes
- Estimating AI costs by number of requests instead of number of tokens per request.
- Sending whole documents with every request when only a short excerpt is needed.
- Forgetting that the model's reply counts towards the limit and the bill as well as the question.
Questions people ask
How many words is 1,000 tokens?
In English, 1,000 tokens is roughly 750 words, but it varies with the text. Technical terms, names, numbers and code use more tokens per word, and many non-English languages use noticeably more. Some providers offer a tokeniser page, and APIs report the exact count for each request.
Why do AI tools charge per token?
Because tokens measure the actual work the model does. Every token read or written takes computing time on expensive hardware, so charging per token ties the price to the effort. Consumer chat apps usually hide this behind a monthly plan with usage limits, while APIs show it directly.
Does a token mean the same as a word?
No. A token can be a whole word, part of a word, a punctuation mark or a space attached to a word. Short common words are usually one token, while a long word like "unsubscribing" may be split into several tokens, and the exact split differs between tokenisers.
