learn me a gpt
Pages in Beginners

19 Nov 2026

Text becomes numbers

Each token id selects a vector from an embedding table, and GPT-2 adds a second vector that depends on the position.

A model can only calculate with numbers, so each token must become a list of numbers before anything else happens. Each token id selects one vector from a big table called the embedding table, and GPT-2 then adds a second vector that depends on the token's position in the text.

A token id is the whole number the tokenizer gives each piece of text. A vector is an ordered list of numbers. An embedding is the vector that stands for a token.

The id is an index into a table

An index is a row number. The embedding table has one row for every token in the vocabulary, and the token id says which row to read. Rows are counted from 0, so id 0 is the first row.

In the tiny example, each row holds 3 numbers:

Token idTokenRow
0the(2, 0, 1)
1cat(1, 3, 0)
2sat(0, 1, 4)

GPT-2 small's table is much bigger. It has 50257 rows, one for each token in its vocabulary, and each row holds 768 numbers. Its shape is 50257 by 768.

The looked-up row is the token vector

Looking up a token takes two steps:

  1. Take the token id, for example 1 for "cat".
  2. Read row 1 of the table: (1, 3, 0).

That row is the token vector. In code the selection is written row = table[id], which means "row is the row of the table at number id".

The id only says where to look. It is not an amount. Id 2 is not twice as much as id 1, and "sat" is not bigger than "cat". Think of numbered lockers: locker 200 is not larger than locker 100, it is just in a different place. The comparison stops at what is inside: every locker here holds a list of numbers of exactly the same length.

The same token always reads the same row. "cat" gets (1, 3, 0) whether it is the first word or the hundredth.

GPT-2 adds a learned position vector

If only the token vector were used, "cat sat" and "sat cat" would turn into the same two vectors in a different order, with nothing inside each vector saying where it stood. So GPT-2 keeps a second table, the position table. It has one row for each position from 0 through 1023: 1024 rows, each holding 768 numbers. Its shape is 1024 by 768.

The tiny example has two positions:

PositionRow
0(1, 1, 0)
1(0, 2, 1)

For each token in the text, GPT-2 does three steps:

  1. Look up the token vector, using the token id.
  2. Look up the position vector, using the token's position. The first token is position 0.
  3. Add the two vectors, position by position.

For the text "cat sat":

  1. "cat" is id 1 at position 0: (1, 3, 0) + (1, 1, 0) = (1 + 1, 3 + 1, 0 + 0) = (2, 4, 0).
  2. "sat" is id 2 at position 1: (0, 1, 4) + (0, 2, 1) = (0 + 0, 1 + 2, 4 + 1) = (0, 3, 5).
Token tablePosition tableSumthecatsat01201130014110021240+=id 1, "cat"GPT-2: 50257 rowsposition 0GPT-2: 1024 rowsblock inputGPT-2: 768 numbers
Token vector plus position vector

The numbers in both tables are learned parameters. A parameter is a number inside the model that training sets, rather than a person writing it by hand. The original transformer paper from 2017 did something else: it computed its position vectors from sine and cosine waves, a method called sinusoidal positions. GPT-2 does not compute its position vectors. It stores them in the learned table and looks them up like token vectors.

The position table stops at row 1023, so GPT-2 small has no position vector for a 1025th token. This matches its context window: it reads at most 1024 tokens at once.

The code below builds the two tiny tables, looks up "cat" at position 0 and "sat" at position 1, and adds each pair of rows position by position.

# Real GPT-2 small shapes:
#   token table:    50257 rows x 768 numbers  (50257×768)
#   position table:  1024 rows x 768 numbers  (1024×768)
# The tiny tables below use 3 tokens, 2 positions, and 3 numbers per row.

token_table = [
    [2, 0, 1],  # id 0: "the"
    [1, 3, 0],  # id 1: "cat"
    [0, 1, 4],  # id 2: "sat"
]

position_table = [
    [1, 1, 0],  # position 0
    [0, 2, 1],  # position 1
]

def add(a, b):
    # Add two rows position by position.
    return [x + y for x, y in zip(a, b)]

ids = [1, 2]  # "cat sat"

for position, token_id in enumerate(ids):
    token_vector = token_table[token_id]         # row = table[id]
    position_vector = position_table[position]
    print(add(token_vector, position_vector))

# prints:
# [2, 4, 0]
# [0, 3, 5]

That sum enters the first block

The sum of the token vector and the position vector is the block input. In GPT-2 small, every token in the text ends up as one such vector of 768 numbers.

A block is one stage of calculation, and GPT-2 small has 12 of them stacked one after another. The list of block input vectors, one per token, goes into the first block. What happens inside the blocks is the subject of later pages.