Tokens: how a chatbot cuts your text into pieces before it reads a word

How does a chatbot read a sentence?

You see letters and words. A language model does not. Before it does anything else, a small program called a tokenizer chops your text into pieces, and each piece gets a number. Those pieces are called tokens. The model never sees the letters of your question, only the list of numbers for its tokens.

A token is not always a word. It can be a whole common word, part of a rare word, a single letter, a space joined to a word, or a piece of punctuation. This matters for you because many chatbots measure their limits in tokens (how much text fits in one conversation) and many services charge by the token. Knowing what a token is explains why a short sentence in one language can cost more than a longer one in another.

Why not just use letters, or just use words?

There are two obvious ways to cut text, and both have a problem.

One token per letter. The list of possible tokens is tiny, so nothing is ever unknown. But text becomes very long. A sentence of 100 characters would be 100 tokens, and the model has to learn from scratch that c, a, t in a row means a furry animal.

One token per word. Text stays short, but the list of possible words never ends. People invent words, make typos, add numbers and names. A model with a fixed list has no token for unbelievably if it never saw that exact word, so it would have to give up.

The common middle path is subwords: frequent words get one token, and rare words are built from smaller pieces. Nothing is ever unknown, because in the worst case you fall back to single letters (or bytes). The question is how to decide which pieces deserve their own token.

Byte pair encoding: merge the most common pair

One widely used answer is byte pair encoding, or BPE. It began as a data compression trick in the 1990s, and researchers adapted it for splitting text into subwords for language models in the mid-2010s. The recipe is short:

1. Start with every letter as its own token.
2. Count every pair of neighbouring tokens in your training text.
3. Take the pair that appears most often and glue it into one new token.
4. Repeat from step 2, as many times as you like.

Each repeat adds one new piece to the vocabulary (the list of all tokens) and makes the text a little shorter. Here is the counting step in Python, on a tiny text:

from collections import Counter

text = "the cat sat on the mat"
pairs = Counter()
for word in text.split():
    for a, b in zip(word, word[1:]):
        pairs[a + b] += 1

print(pairs.most_common(3))

Running it prints:

[('at', 3), ('th', 2), ('he', 2)]

The pair a + t shows up three times (in cat, sat and mat), so BPE would make at its first new token. Nobody told it that at is meaningful. It simply noticed that those two letters travel together.

Try it: train a tokenizer yourself

Below, a real (very small) BPE trainer runs in your browser. Press Merge next pair and watch the pieces grow. The highlighted pieces are the pair about to be glued together. A small dot · stands for a space. You can also replace the text with your own sentence.

With the starting sentence, seven merges shrink 44 single-letter pieces to 21. The first new piece is at, then th, then the. After that the trainer even learns a piece for a space followed by the. Try a sentence with no repeated letter pairs and nothing merges, because there is nothing worth gluing. Try pasting the same word ten times and watch it collapse into just one or two pieces.

What a real tokenizer adds

The trainer above is the same idea real tokenizers use, at toy size. Real ones differ in scale and detail. They are trained on enormous amounts of text, and their vocabularies usually hold tens of thousands to a few hundred thousand pieces, so most common English words become a single token and rare words, names and typos are split into several. Different models use different tokenizers, and some use other methods than BPE, so the same sentence can have a different token count in different models.

Because the vocabulary is learned from the text it was trained on, text that was less common in that training data tends to be cut into more, smaller pieces. That includes many non-English languages, unusual names, long numbers and code with unusual spelling. More pieces means more tokens used for the same meaning.

So what, for your own code?

If you build something that calls a language model, remember three things. First, limits and prices are usually counted in tokens, not characters or words, so use the tokenizer or token counter your provider offers instead of guessing from word count. Second, tokens explain some odd model behaviour: a model sees pieces, not letters, so questions like "how many times does the letter r appear in this word" are harder for it than they look. Third, a rule of thumb like "a token is roughly three quarters of an English word" is only an average. It can be far off for other languages or for code.

The idea to keep: a language model reads text as a list of learned pieces, and those pieces come from counting which neighbours appear together most often.