AI Terminology Course
AI Terminology
/
Beginner

Tokens

Definition

The fundamental units of data processed by an LLM. Depending on the tokenizer used, a token can be an entire word, a syllable, or just a single character.

Explain Like I'm New

The 'coins' of the AI world. You pay for AI by the token. Rule of thumb: 1 Token is approximately 4 characters of English text, or about 3/4 of a word.

Real World Example

If you submit a 1,000-word essay to the OpenAI API, you will be billed for roughly 1,300 tokens.

Common Use Cases

  • •API billing
  • •Model context limits

Interview Questions

basic

  • Roughly how many English words is 1,000 tokens?

intermediate

  • Why does generating code (like Python or JavaScript) usually consume significantly more tokens than generating plain English text?

Flash Cards

Question

How many words?

Click to reveal answer
Answer

Roughly 750 words.

Question

Why does code consume more?

Click to reveal answer
Answer

Because code contains a massive amount of special characters, weird variable names, and excessive spaces/indentation. A tokenizer cannot group `my_weird_var()` into a single word, so it splinters it into 5 or 6 separate tokens.