How LLMs "Talk": Tokens, Context, and Probability

ChatGPT looks like a person typing, but the machinery underneath is different. Three concepts unlock Prompt, RAG, and Agent later.

Tokens: The Model’s Alphabet

Models do not process “characters” or “words” directly—they process tokens, units somewhere between characters and words.

"artificial intelligence" → might split as ["art", "ificial", " intelligence"]
"ChatGPT"               → ["Chat", "G", "PT"]

Why it matters:

• Billing is per token: input + output tokens = cost
• Context limits are in tokens: “128K context” = up to 128,000 tokens
• Rough rule: ~1–2 tokens per English word; CJK often ~1–2 tokens per character

Context Window: Short-Term Memory

Everything you send—system prompt, chat history, current question—lives in the context window.

┌─────────────────────────────────┐
│  System prompt                  │
│  Conversation history           │
│  Your current question          │
│  ← model generates from here →  │
└─────────────────────────────────┘
        Context window (limited)

Key facts:

• When full, oldest content gets pushed out—the model “forgets”
• Window sizes vary: 4K, 32K, 128K, 200K…
• Larger windows fit more docs but cost more and run slower

Probabilistic Sampling: Guessing One Token at a Time

Generation is repeatedly predicting the next token:

Input: "The weather today"
Next-token probabilities:
  "is"      → 35%
  "looks"   → 28%
  "feels"   → 15%
  "will"    → 12%
  ... other → 10%

Then sample one token. Two important knobs:

• Temperature: higher = more random/creative; lower = more deterministic
  • Code → low (0–0.3)
  • Brainstorming → high (0.7–1.0)
• Top-p: sample only from the top p% of candidates

Why AI Hallucinates

The model is not looking up facts—it predicts text that sounds true. Frequent facts in training get reinforced; missing facts may be invented plausibly.

That is mechanism, not a bug. So:

Never let an LLM alone handle tasks requiring 100% factual accuracy—unless paired with RAG or tool verification (covered later).

Try It Yourself

Next time you use ChatGPT, notice:

1. Long chats—does it “forget” earlier details?
2. Same question twice—slightly different answers? (sampling randomness)
3. Hard math—does it err? (prediction vs. calculation)

Summary

Concept	One line
Token	Smallest unit; drives billing and capacity
Context window	Short-term memory with a hard cap
Sampling	Token-by-token prediction; temperature controls randomness

Next: now that you know how the model works—how do you ask so it answers well? Prompt basics.