Training is the stage where a model's stored numbers are adjusted, so that its next-token predictions match real text. Inference is the stage where those stored numbers are used to produce output, without changing them.
Tuning a guitar and playing it is a fair comparison. Tuning changes the strings. Playing uses the strings as they are. The comparison stops at the method: a guitar is tuned by ear, while training changes the numbers by calculation, using a very large amount of text.
Inference is the forward pass
A forward pass is one run of the model as a function: token ids go in, and a distribution over the next token comes out. Inference is running forward passes and choosing tokens from what they return.
Take the text "The cat sat on the":
- Its token ids go in: 464, 3797, 3332, 319, 262.
- The model reads its parameters, the stored numbers that define the calculation.
- A distribution comes out: one probability for every possible next token.
- One token is chosen from it, for example " mat".
At step 2 the parameters are only read. Nothing writes to them. After the pass they hold exactly the same values as before.
Training adds a comparison and an update
Training needs text where the answer is already known. Every position in a sentence is a small test. In "The cat sat on the mat", the text itself shows that "The" is followed by " cat", "The cat" by " sat", and "The cat sat on the" by " mat".
Here is one training step for that last position:
- Forward pass. "The cat sat on the" goes in, and a distribution comes out. Suppose it gives " mat" a probability of 0.05. That number is made up for illustration.
- Comparison. The training text says the next token was " mat". The comparison measures how much probability the model gave to that real next token. The result is one number, called the loss. A low probability for the real token means a high loss.
- Update. The parameters are changed by a small amount, in the direction that would give " mat" a little more probability after "The cat sat on the".
- Repeat. The same steps run on a very large number of positions, across a large amount of text.
After many updates, the distributions the model produces come to match the patterns in the training text.
For GPT-style models, the comparison is a next-token loss called cross-entropy. Its formula is on that page. The method that decides how much to change each parameter is also left to later pages.
A chat product is inference
When you use a chatbot, each reply is built from many forward passes in a row. Each pass returns a distribution, one token is chosen, and the next pass runs with that token added.
There is no comparison and no update on this path. The parameters are the same at the end of a conversation as at the start. The chatbot can refer to earlier messages only because the product sends the earlier transcript again with each request.
The same forward calculation appears in both
Training and inference share the same forward pass. They differ in what happens after it:
- Inference: forward pass, then a choice of token. It stops there.
- Training: forward pass, then the loss, then an update to the parameters.
Because the calculation is shared, the distributions you see during a chat come from the very calculation that training adjusted. Training shaped the parameters. Inference only reads them.