This tool splits your text into GPT-2 tokens and shows each token’s text and its id number.
The splitting runs in your browser, and the text is not sent anywhere.
Up to 200 characters. 0 / 200 used.
Tokens:0
Type some text to see its tokens.
The text is split by the GPT-2 tokenizer in this browser and is not sent anywhere.
The Python below shows the same call with the tiktoken library. It is the same GPT-2 mapping from text to ids as the browser tool, not a second algorithm, so it gives the same ids for the same sentence.
import tiktoken
enc = tiktoken.get_encoding("gpt2")
ids = enc.encode("A cat sat on the mat.")
print(ids) # [32, 3797, 3332, 319, 262, 2603, 13]
print(enc.decode(ids)) # A cat sat on the mat.