learn me a gpt
Pages in Beginners

19 Nov 2026

GPT-2 and later models

GPT-2 small is a public transformer this site can open. Later models keep that stack and change named pieces. A chat product adds training and serving around a model.

GPT-2 small is a public transformer this site can open. Later models keep that stack and change named pieces. A chat product adds training and serving around a model.

Put plainly: GPT-2 small is a model whose file anyone can download and read. Newer models are built the same basic way, with a few parts swapped. A chat app you use is a model plus extra work wrapped around it.

What is public in GPT-2

The weights and code for GPT-2 small are public. Weights are the model’s parameters, the numbers stored in its file. The code is the program that runs the forward pass with those numbers.

Because both are public, every number can be looked at. The sizes are listed on the GPT-2 config page: 12 layers, hidden size 768, 12 heads, and a vocabulary of 50257 tokens. The context length is 1024, so GPT-2 small reads at most 1024 tokens at once.

The site’s explorer does not run GPT-2 small while you watch. It shows one traced sentence. A trace is a saved record of one run:

  1. One sentence is split into tokens.
  2. GPT-2 small runs its forward pass on those tokens once.
  3. The numbers at each step are written to a file.
  4. The explorer reads that file and draws it.

So the explorer is a recording, not a live service. It cannot answer a new message.

A chat product is not the base model

A base model is the model as it comes out of its first training. It continues text. Give it the start of a sentence, and it scores possible next tokens.

A product such as ChatGPT adds three things around a model:

  1. Further training. More training teaches the model to reply in turns, as an assistant.
  2. Serving. Computers run the model for many people at once and send back the replies.
  3. A transcript UI. UI means user interface, the screen you type into. Here it is the chat window that shows your messages and the replies as one conversation.

You can compare a base model to an engine on a test stand, and a chat product to a car built around it. The comparison breaks in one place: further training changes the model’s own parameters, so the engine itself is altered, not only surrounded.

This site does not load that product’s weights. Closed chat models are described here and are not inspected. Only GPT-2 small is opened.

Later open models change specific pieces

An open model is one whose weights are published. Many later open models keep GPT-2’s overall stack:

  1. Text becomes tokens.
  2. Each token becomes a row of numbers.
  3. The rows pass through a pile of repeated blocks, each with attention and a feed-forward step.
  4. The last rows become scores for the next token.

What changes is a short list of named pieces inside that stack. Each one has its own page.

  • Rotary position. GPT-2 adds a row from a position table to each token. Rotary position instead turns the numbers that attention compares, by an amount set by each token’s place.
  • RMSNorm. A simpler version of the layer-norm step, which keeps each row in a steady range.
  • A gated feed-forward, also called a gated MLP. The feed-forward step gets an extra path that acts like a gate, deciding how much of each number passes through.
  • Grouped-query attention. Attention is split into heads. Here several heads share one set of the rows that attention looks up, so fewer of those rows are made and kept.

These are changes to parts, not a new kind of system. The model still reads tokens and scores the next one.

GPT-2Later open modelToken tablePositionLayer normAttentionFeed-forwardNext-token scoresToken tableRotary positionRMSNormGrouped queryGated MLPNext-token scoresThick boxes are the changed pieces
Later models change named pieces of the same stack.

The technical section continues from here

The beginner pages end here. The technical section explains the same system one calculation at a time. Its order starts at vectors and reaches these later pieces at the end.

The map lists every page in that order, including pages that are not written yet.