learn me a gpt
Pages in Beginners

19 Nov 2026

Tokens

A token is an item in a fixed vocabulary, and a piece of text is a sequence of those items’ ids.

A token is one piece of text from a fixed list of pieces the model knows. That list is called the vocabulary, and every entry in it has a number, its id. To a GPT, your message is not letters. It is a sequence of those ids.

Text is split before any layer runs

A GPT only computes with numbers. Its layers, the stages of calculation inside the model, never see a letter. So before any layer runs, a separate program called a tokenizer turns your text into a list of integers (whole numbers).

Here is what GPT-2’s tokenizer does with the sentence “The cat sat on the mat.”

  1. It takes the text as one string. A string is a programmer’s word for a piece of text.
  2. It cuts the string into pieces found in the vocabulary: The, cat, sat, on, the, mat, and .. The space in front of a word belongs to that word’s piece.
  3. It replaces each piece with its id. The result is a list of seven integers: 464, 3797, 3332, 319, 262, 2603, 13.

That list of seven numbers is everything the model receives. The sentence itself stays outside.

A vocabulary works a little like a numbered dictionary: look up an entry, get its line number. The comparison stops at the entries. A dictionary lists whole words, and a vocabulary also lists word pieces, single characters, and punctuation.

GPT-2 uses a byte-level vocabulary of 50257

GPT-2’s vocabulary has exactly 50257 entries. Their ids run from 0 to 50256. The list was fixed before the model was trained, and it never changes while you use the model.

The vocabulary is byte-level. Computers store every character as one or more bytes, and a byte is a small number from 0 to 255. Every possible byte has its own entry in the vocabulary. Most of the remaining entries are common chunks of text, such as a whole word or a space followed by a word.

Because single bytes are always available, any text can be encoded. That includes any language, any symbol, and any emoji that Unicode (the standard numbering for the world’s characters) describes. When a chunk is not in the list as a whole, the tokenizer uses smaller pieces, down to single bytes if needed. Nothing is ever “unknown.”

A word is not always one token

Common words are usually one token. Rarer words, names, and long words are often split into several. These splits were computed with GPT-2’s tokenizer:

  • cat is one token.
  • tokenization is two tokens: token and ization.
  • ChatGPT is three tokens: Chat, G, and PT.
  • antidisestablishmentarianism is five tokens: ant, idis, establishment, arian, and ism.

The pieces do not have to match meaningful parts of a word. In the last example, ant and idis are not the word parts a dictionary would list. Tokens do not match English words, and they were never meant to.

Small details change the split too. Hello at the start of a text and hello after a space are two different tokens, with two different ids. Capital letters, spaces, and punctuation all count.

textI love tokenization!tokensIlovetokenization!idsididididid
Text becomes a sequence of vocabulary ids

Up to 200 characters. 20 / 200 used.

Tokens:5
  1. Iid 40
  2. ·loveid 1842
  3. ·tokenid 11241
  4. izationid 1634
  5. !id 0

Each box is one token: its text on top, its token id underneath. A · stands for a space. Many tokens begin with a space because GPT-2 keeps the space together with the word after it.

Token ids:40, 1842, 11241, 1634, 0

The text is split by the GPT-2 tokenizer in this browser and is not sent anywhere.

The ids are indexes

An index is a position in a list. Id 3797 means “entry number 3797,” and nothing more. It is an address, not an amount.

So the size of an id says nothing about the token. Id 17 is not more, bigger, or more important than id 16. Two ids that are close together do not have to mean similar things. The number 17 is not the meaning of a token.

What the id does is pick a row. Inside the model is a table with one row for each of the 50257 tokens. Each row is a list of numbers that the model learned during training; in GPT-2 small, each row holds 768 numbers. This row is called the token’s embedding, and it is what the layers compute with.

Think of locker numbers. Locker 17 is not worth more than locker 16; the number only says which door to open. The comparison stops at neighbours. Lockers 16 and 17 sit side by side, and tokens 16 and 17 have no such connection.