learn me a gpt
Pages in Beginners

19 Nov 2026

Parameters

A parameter is a number stored in the model. The forward pass reads the set, and training writes it.

A parameter is a number stored in the model. The forward pass reads the set, and training writes it.

The forward pass is the one calculation that turns your tokens into scores for the next token. Every step of that calculation multiplies or adds stored numbers. Those stored numbers are the parameters. GPT-2 small keeps more than a hundred million of them in one file.

You can think of parameters as the settings saved in a file. The comparison stops there: a person picks a setting, but no person picks these numbers. Training sets them, and a single parameter on its own rarely means anything you could name.

Name the main stores

The parameters are kept in a few kinds of store. Each store is a grid of numbers, or a single list of numbers.

  1. Token table. One row of numbers for every token in the vocabulary. The vocabulary is the fixed list of tokens the model knows.
  2. Position table. One row of numbers for every position in the input: first token, second token, and so on.
  3. Attention projections, in each block. Grids that turn a token’s row into the pieces attention compares and mixes.
  4. Feed-forward projections, in each block. Two grids that widen each token’s row, then narrow it back.
  5. Layer-norm scales and shifts, in each block. Short lists that rescale a row and nudge it up or down. A layer norm is a step that keeps the numbers in a steady range.

Here is how the first two stores are used when you type text.

  1. The text is split into tokens.
  2. For each token, the model looks up its row in the token table.
  3. For each position, the model looks up a row in the position table.
  4. The two rows are added together, number by number.
  5. The result goes into the first block, which uses that block’s attention, feed-forward, and layer-norm parameters.
  6. The same happens in each block after it, each with its own parameters.

The exact name and shape of every stored grid is listed on the weight names page. This page only needs the main kinds.

Token tablePosition table50257 rows × 7681024 rows × 768One block× 12Attention weightsFeed-forward weightsLayer-norm scales and shifts
Parameters are the stored tables and the block weights.

The shape fixes the count

The number of parameters is not a separate choice. It follows from a few size settings, called the model’s config. GPT-2 small uses these:

  • 12 layers. There are 12 blocks, stacked one after another.
  • Hidden size 768. Every token’s row holds 768 numbers.
  • 12 heads. Attention in each block is split into 12 parallel parts, called heads.
  • Context 1024. The model reads at most 1024 tokens at once, so the position table has 1024 rows.
  • Vocabulary 50257. There are 50257 tokens, so the token table has 50257 rows.
  • Feed-forward width 3072. The feed-forward step widens each row to 3072 numbers, which is 4 × 768.

Once the sizes are fixed, the size of each grid is fixed too. You can work out single stores with plain multiplication:

  • Token table: 50257 rows × 768 numbers = 38,597,376 numbers.
  • Position table: 1024 rows × 768 numbers = 786,432 numbers.
  • One feed-forward grid that widens a row: 768 × 3072 = 2,359,296 numbers.

These are single stores, not the model’s total. Change one size setting, and the grids that use it grow or shrink with it.

Published totals differ by what they count

The GPT-2 paper calls the small model 117M, meaning about 117 million parameters. Reports made from the released checkpoint are often near 124M instead. A checkpoint is the saved file of parameters.

The gap comes from how embeddings are counted. An embedding is a row looked up from the token table or the position table. The token table is also reused at the output end of the model, to score every possible next token, and counts differ in how they treat these tables.

Both labels name the same model. This page does not give a total of its own. The config page computes a count from a loaded checkpoint and records what it included.

Inference reads them as constants

Training changes the parameters, and inference, the use of a finished model, only reads them. When you send text to GPT-2 small, every parameter stays exactly as it was in the file. The same input with the same parameters gives the same next-token scores.