Tokens and subwords

Before a text goes into a language model, it is split into pieces. Each piece is called a token, and those are the pieces the model then reads.

What a token is

A tokenizer does the splitting. It has a fixed vocabulary, a list of pieces it is allowed to use. A common word is often a single token. An unusual or long word is split into several shorter pieces. Those pieces are called subwords.

Each model family, meaning each group of related models, has its own tokenizer, so the same text can come out as a different number of tokens in different models.

For English text, one token is on average about four characters, about three-quarters of a word. Swedish usually comes out as more tokens per word than English, because the vocabularies were built mostly on English text.

Why words are split

Words are split because the vocabulary is fixed and only holds certain pieces. A common word often has a piece of its own. When a word is unusual or long, it is split into several subwords that are already in the vocabulary.

The exercises use a small made-up vocabulary, so the split has only one valid answer. A tokenizer then always takes the longest piece from the left that is in the vocabulary and continues with the rest of the word. The pieces are in the box beside the text.

Why it costs and takes up space

Tokens matter for two things. The first is price. APIs, the interfaces programs use to send text to a model, charge per million tokens. The more tokens the text becomes, the higher the price. Swedish usually comes out as more tokens per word than English, so a Swedish text often costs more than an English text with the same number of words.

The second is the model's context window. That is how much text the model can see at once, and it is measured in tokens. All the text that needs to fit in the same go has to fit inside that number. If you count the words, you can think that more text will fit than the window holds, because the window is counted in tokens.

What you should be able to do after the lesson

• Explain that a tokenizer with a fixed vocabulary splits text into tokens, that a common word is often a single token, and that an unusual or long word becomes several subwords.
• Explain that the same text can come out as a different number of tokens in different model families, and that Swedish usually takes more tokens per word than English.
• Explain that the price of an API is charged per million tokens and that the context window is measured in tokens.