Skip to content

Tokens -- The Building Blocks of Language

  • Beginner · Module 03
  • Lesson 2 of 7 · then sources and quiz

Step 1 of 4Knowledge

When you talk to an LLM, it does not read your message word by word but breaks it down into smaller units: tokens. A token is a subword unit -- roughly three quarters of a word. Some short words like "AI" or "is" are a single token. Longer words like "Artificial" are broken into 2-3 tokens.

Why not just use whole words? Because every language has millions of possible words (think of compound words like "Donaudampfschifffahrtsgesellschaft" in German). By breaking text into tokens, the model can work with a manageable vocabulary of 30,000-100,000 units.

Step 2 of 4Understanding

To give you a sense of token quantities, here are some everyday comparisons:

  • A text message: about 25-30 tokens
  • A short email: about 250-300 tokens
  • A newspaper article: about 1,000-2,000 tokens
  • A novel: about 100,000 tokens

Tokens matter for a practical reason: costs are calculated per token. Every token you send to an LLM (input) and every token it sends back (output) costs money. Additionally, there is a token limit -- the maximum number of tokens for input and output combined.

Tip
What Does That Cost in Practice?

With common LLMs, 1,000 tokens cost about 0.001 to 0.01 euros. A typical conversation with 10 messages costs only a few cents. With free offerings like ChatGPT Free or Claude Free, you pay nothing -- the provider covers the costs.

Interactive · Token breakdown
ArtificialIntelligenceischangingourworld
14
Tokens
45
Characters
$0.00028
Approximate cost (large model, output)
Note: This is a simplified simulation. Real tokenizers (like BPE) work differently.

Step 3 of 4Apply

If you use an LLM and get a very long response that breaks off mid-sentence, you now know why: the token limit was reached. You can simply write "Please continue" and the LLM will pick up where it left off.

Exercise · Fill in the gaps
Estimate the approximate token count:
A text message has approximately tokens.

Step 4 of 4Reflect

Question · Self-check
Why are costs for LLMs calculated per token and not per word?