Beginners
This reading order starts from a chat window and ends at the shape of a GPT-2 model.
- How a GPT works
A GPT-style model repeats one step: it turns the current tokens into a distribution over the next token, appends one token, and repeats.
- Using a chatbot
A chatbot sends the conversation so far to a model and appends the model’s new tokens to that conversation.
- Prompts
A prompt is the token sequence already present when generation starts.
- Tokens
A token is an item in a fixed vocabulary, and a piece of text is a sequence of those items’ ids.
- The context window
The context window is the maximum number of tokens the model will take at one step.
- Likelihood and truth
The training target is the next token in the text, so a fluent continuation is not a check that the continuation matches the world.
- What a chat request contains
A chat request contains the text the product packs for the model, and a hosted product processes that text on the operator’s computers.
- A model is a function
During a reply, the model is a fixed function from the current token ids to a distribution over the next id.
- Training and inference
Training changes parameters so the next-token distributions match the training text. Inference uses the parameters and does not change them.
- Vectors
A vector is an ordered list of numbers. The dot product multiplies matching positions and adds those products.
- Text becomes numbers
Each token id selects a vector from an embedding table, and GPT-2 adds a second vector that depends on the position.
- The next token
The model produces one score per vocabulary id, and softmax turns those scores into probabilities that sum to 1.
- Attention
Attention builds a new vector for each token as a weighted sum of value vectors from tokens it is allowed to read.
- The transformer
A GPT-2 transformer is a stack of blocks. Each block mixes tokens with attention and then passes each position through the same feed-forward network.
- Temperature
Temperature divides the logits by a positive number before softmax. Above 1 the distribution flattens. Below 1 it sharpens.
- Parameters
A parameter is a number stored in the model. The forward pass reads the set, and training writes it.
- GPT-2 and later models
GPT-2 small is a public transformer this site can open. Later models keep that stack and change named pieces. A chat product adds training and serving around a model.