learn me a gpt
Pages in Beginners

19 Nov 2026

The transformer

A GPT-2 transformer is a stack of blocks. Each block mixes tokens with attention and then passes each position through the same feed-forward network.

The transformer is the main body of a GPT: a pile of layers that the token vectors pass through, one layer after another. A GPT-2 transformer is a stack of blocks. Each block mixes tokens with attention and then passes each position through the same feed-forward network.

Follow the prompt "The cat sat" through it. The prompt is three tokens, and each token has already become a vector of 768 numbers. So the transformer starts with three rows of numbers, one row per token, each row 768 numbers long.

One block does two jobs

A block is one layer of the stack. It takes rows of numbers in and gives the same number of rows out, each still 768 numbers long. Inside, it does two jobs in order:

  1. Attention mixes across positions. The row for "sat" takes in shares of the rows for "The" and "cat", as described on the attention page. This is the only step where rows exchange information.
  2. The feed-forward network transforms each position on its own. A feed-forward network is a fixed calculation with learned numbers. It stretches one row from 768 numbers to 3072, applies a fixed curve to each number, and shrinks the result back to 768.

The feed-forward step uses the same learned numbers at every position. The row for "The" and the row for "sat" go through exactly the same calculation. No row looks at another row during this step.

A meeting followed by desk work is a useful comparison. In the meeting, each person hears from the people who spoke before them. Then everyone goes back to a desk and works alone, following the same instructions. The comparison stops at the people: a block reads no meaning, it only does arithmetic on rows of numbers.

GPT-2 small stacks 12 blocks

GPT-2 small has 12 blocks. The output of one block is the input of the next. For "The cat sat":

  1. The three tokens become three rows of 768 numbers.
  2. Block 1 turns them into three new rows of 768 numbers.
  3. Block 2 takes those rows and produces three more rows of 768 numbers.
  4. The same happens in blocks 3 to 11.
  5. Block 12 produces the final three rows of 768 numbers.

The length of each row is called the hidden size. In GPT-2 small it is 768 at the start of the stack, between every pair of blocks, and at the end. Only the numbers inside the rows change.

All 12 blocks have the same shape, but each block has its own learned numbers. Block 5 does not reuse the numbers of block 4. Because each block works on rows that earlier blocks have already mixed, information can travel further with every block.

The cat sat: 3 rows of 768Block 1Block 2Block 3blocks 4 to 11Block 12Logits: 50257 scoreseach block:attention,thenfeed-forwardeach rowstays 768
GPT-2 small stacks 12 blocks.

A final step produces logits

After block 12, one more steadying step runs, and then a final calculation turns a row of 768 numbers into 50257 numbers. GPT-2 has a vocabulary of 50257 token ids, so there is one number for each id. These numbers are the logits: scores that softmax turns into probabilities for the next token.

The final calculation is a linear map. Each logit is made by multiplying the row's 768 numbers by 768 learned numbers, one pair at a time, and adding the products. The row at the last position, "sat", gives the scores for the token that comes after "sat".

GPT-2 does not store a separate table of learned numbers for this step. It reuses the token embedding: the table that turned each token id into its starting vector of 768 numbers. That one table is used twice, once at the start to look up each token's vector and once at the end to score every possible next token. The stored names of these tables are listed on a later page.

The same stack runs at every generation step

How a GPT works describes generation as a loop: tokens go in, a distribution over the next token comes out, and one token is appended. Every turn of that loop runs the whole stack again, all 12 blocks and the final step, with the same stored numbers, on the prompt plus every token added so far.