Clicked Gallery

What is Tokenization?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

Engineering Notes · AI Systems

During preprocessing, tokenization splits the raw text into subword units the model actually reads.

The reader highlighted one word in the docs. Clicked explained the technical term “tokenization” in simple terms:

Explained in three depths

Same facts, different vibe — Slang mode 😎

The Clicked way

●○○

Overview

Tokenization is how AI models chop text into pieces before reading it. The pieces, called tokens, are usually chunks of words rather than whole words. Models see, count, and charge by these tokens, not by words or letters.
●○○

Overview

Tokenization is the AI chopping your text into bite-size chunks called tokens before it reads anything. Not letters, not words, something in between — and it's the only alphabet the model has. 😎

A quick take — often all you need.

●●○

Detail

A tokenizer carries a fixed vocabulary, typically 50,000 to 200,000 entries, built by finding the character combinations that appear most often in training text. Common words such as “the” survive as single tokens, while rarer ones get split, so unhappiness might become un, happi, and ness. In English this averages out near three-quarters of a word per token, but the same sentence in Japanese or Hindi can cost two or three times more tokens, since the vocabulary was built mostly from English text. That gap matters because API pricing is per million tokens, so the identical question is literally more expensive in some languages. Tokenization also explains a famous failure: a model that can't count the letters in “strawberry” isn't seeing letters at all, it's seeing two or three chunks. Numbers get chopped at odd places too, which is one reason arithmetic trips models up.
●●○

Detail

The tokenizer owns a fixed menu of 50k to 200k text chunks, learned from whatever it trained on, and every sentence you send gets rebuilt from that menu. Everyday words ride as one token while weird ones get diced, un-happi-ness style. English averages about three-quarters of a word per token, but the same question in Japanese can burn two or three times more, and since billing is per million tokens, some languages literally pay a language tax. This is also why the “strawberry” thing happens: the model never sees r-r-r, it sees a couple of chunks with no letters inside. Same deal with math, numbers get sliced at random spots, and then everyone acts surprised it can't add. 😎

Want more? One click digs deeper.

●●●

Analogy

Building with LEGO. A castle isn't molded in one piece; it's assembled from a fixed catalog of standard bricks, and the builder only ever thinks in bricks. Ask about the plastic grain inside one brick and you get a blank stare, because the brick is the smallest thing that exists to them.
Deepest level reached.
●●●

Analogy

Emoji-only texting. You can say a lot with the standard set, but only what the set covers. There's no emoji for your specific weird Tuesday, so you approximate with three that kind of gesture at it, and some things just cannot be said.
Deepest level reached.

Unfamiliar concept? A real-world example makes it click — fresh analogies on tap.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

Tokenization is the segmentation of input text into discrete units drawn from a fixed vocabulary, typically constructed via byte-pair encoding or similar subword algorithms that merge frequent character sequences. Token counts determine sequence length, context consumption, and usage-based billing, and tokenization granularity varies substantially across languages and scripts.

Want Clicked to explain terms like “tokenization” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required