A GPT is a program that continues text. It repeats one step: it turns the current tokens into a distribution over the next token, appends one token, and repeats.
Everything else on this page is detail around that one loop.
A GPT continues text
A GPT does not read text as letters or whole sentences. It reads tokens: small pieces of text, often a whole word and sometimes part of one.
Take the text "The sky is". Here is what happens to it, step by step.
- The text is split into tokens: "The", "sky", "is".
- The model does a long calculation on those tokens. The result is a chance for every token it knows, saying how likely that token is to come next. This list of chances is called a distribution. The chances add up to 100%.
- One token is chosen from the distribution. Likely tokens are chosen more often. Suppose "blue" had a 60% chance and "clear" had 15%. "blue" is the most likely pick.
- The chosen token is added to the end. The text is now "The sky is blue".
- The longer text goes back to step 2, and the model works out the chances for the token after "blue".
The loop stops when the model picks a special token that means "end of text", or when a length limit is reached. A long answer is many passes through the same loop, one token per pass. The same model, with the same stored numbers, runs at every step.
A phone keyboard that suggests the next word works in a similar way: tap the suggestion again and again and it writes a sentence. The comparison stops there. A GPT looks at far more of the earlier text and uses millions of stored numbers to work out its chances.
A chat window is a product around that loop
The model and the chat window are two different things. The model is the calculation above. The chat window is a program built around it.
When you press send, the window collects the conversation so far into one piece of text, called a transcript. The transcript is turned into tokens before the model runs. The model then continues those tokens, and the window displays the new tokens as they arrive. The model never sees buttons or screens, only tokens.
Parameters stay fixed during a reply
The model's calculation uses a large set of stored numbers called parameters. They decide which next tokens get high chances. A reply reads these numbers. It does not change them.
There are two separate processes:
- Training sets the parameters. It happens before the model is used, by running the model on a large amount of text and adjusting the numbers.
- Inference uses the parameters. This is what happens when a reply is generated. The numbers stay exactly as training left them.
Think of a recipe card read while cooking. Cooking a meal does not rewrite the card. The comparison stops there: a recipe is written by a person, while parameters are numbers produced by training.
Attention is the step that mixes tokens
Inside the calculation, each token is held as a list of numbers, and the model updates those lists in several stages. Attention is the stage that lets each token's new list use information from earlier tokens in the text. In "The sky is", the numbers for "is" can take in information from "sky", which is why the model can favour "blue" rather than a word that fits any sentence.
GPT-2 small is the model this site opens
When a page on this site shows a real model's internals, the model is GPT-2 small. Its parameters are public, so every number can be inspected. Its main sizes are:
- 12 layers. The calculation runs through 12 repeated stages, one after another.
- Hidden size 768. Each token is held as a list of 768 numbers.
- 12 heads. In each layer, attention runs 12 times side by side.
- Context length 1024. The model reads at most 1024 tokens at once.
- Vocabulary size 50257. There are 50257 different tokens it can choose from at every step.
Later open models keep the same loop and change specific pieces, such as the number of layers, the context length, or the vocabulary. Closed chat products are described on this site but not loaded, because their parameters are not public.