How Large Language Models Work

By the end of this lesson you'll be able to explain, in plain English and with runnable code, how an LLM turns your prompt into text — token by token.

Learn How Large Language Models Work in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

You already use a tiny language model every day: phone keyboard autocomplete. Type "I'll be there in five" and it suggests "minutes". An LLM is that exact idea — autocomplete on steroids .

Imagine someone who has read almost everything ever written and remembers the patterns. When you type "The cat sat on the", they don't understand cats — they just know that "mat" is the most likely next word. Scale that predictor up to billions of internal settings and trillions of words of practice, and you get something that can write essays, code, and answers. That's all an LLM is doing: predicting the next token, over and over, very well.

An LLM has exactly one job: given some text, output a probability for every possible next token . A token is a chunk of text (often a word piece). The model assigns, say, 62% to "mat" and 21% to "floor", picks one, appends it, and runs again. Generating a paragraph is just this loop repeated hundreds of times — that's called autoregressive generation.

The simplest way to pick is greedy decoding : always take the highest-probability token. Run the worked example below — it takes a probability dictionary and grabs the most likely next word.

GPT, LLaMA, and Mistral all use a decoder-only transformer . The key mechanism is self-attention : at every position, the model looks back at earlier tokens and decides which ones matter for predicting what comes next. "Causal" (or masked) attention means a token can only see tokens before it — never the future. That's what makes left-to-right generation possible.

Reading "The cat sat on" with causal attention:

Stack dozens of these attention-plus-feed-forward layers, and the model can capture grammar, facts, and style. You don't need the matrix maths to use one — but knowing that each token only attends to the past explains why an LLM writes one token at a time.

Models don't read characters or whole words — they read tokens . A rough rule for English is one token per four characters (about ¾ of a word). The context window is the maximum number of tokens — your prompt plus the reply — the model can hold at once. Go over it and the oldest text falls out of view, so the model effectively "forgets" the start of a long conversation.

A model's parameters (also called weights ) are just numbers — billions of them — that the model adjusts during training. They're the dials that turn an input of tokens into an output of probabilities. "GPT-3 has 175 billion parameters" means 175 billion of these numbers.

There's no database of facts inside; the "knowledge" is encoded across all those weights. More parameters generally means more capacity to capture patterns — but, as you'll see in the scaling section, size alone isn't everything.

Greedy decoding always picks the top token, which can feel robotic and repetitive. Instead, models usually sample from the probabilities. The raw scores the model produces are called logits ; softmax turns them into probabilities that sum to 1.

Temperature is the knob. You divide the logits by the temperature before softmax. A low temperature (e.g. 0.5) sharpens the distribution — the top token dominates, output is safe and consistent. A high temperature (e.g. 2.0) flattens it — output gets diverse and creative, but also more likely to go off the rails. Run the example to watch the probabilities shift.

Pretraining is the expensive part: the model learns general language by predicting the next token across trillions of tokens of internet text. This takes months on huge GPU clusters and can cost millions of dollars. The result is a model that knows language but isn't specialised.

Fine-tuning takes that pretrained model and cheaply adapts it to a specific task or style using a much smaller dataset — sometimes just thousands of examples. Techniques like LoRA make this affordable on a single GPU. The rule of thumb: almost everyone fine-tunes; almost no one pretrains from scratch.

The worked examples below use the real Hugging Face transformers library. They need transformers and torch installed locally, so treat them as a preview of the real API — the expected output is shown in comments.

Scaling laws are the surprising finding that a model's prediction error falls in a smooth, predictable way as you add more parameters , more data , and more compute . The Chinchilla result added an important twist: for a fixed compute budget, you often want a smaller model trained on more data — roughly 20 tokens of training data per parameter.

Emergent abilities are skills that barely exist in small models but suddenly appear once a model crosses a size threshold — things like following instructions, doing multi-step arithmetic, or in-context learning. Nobody programmed these in; they emerge from scale. This is why bigger models can feel qualitatively, not just quantitatively, smarter.

Because an LLM optimises for plausible next tokens — not for truth — it will sometimes produce confident, fluent statements that are simply wrong. This is called hallucination . There's no fact-checker inside; if a made-up citation or API method "sounds right", the model may emit it.

The practical fix: verify important facts, ground the model with retrieval (RAG) or tools, and lower the temperature when you need consistency.

❌ Output cut off / "maximum context length exceeded"

Your prompt plus reply went past the context window, so the model truncated or errored.

❌ The model confidently states something false (hallucination)

It generated a plausible-sounding but wrong fact, citation, or function name.

❌ Output is random / repetitive — wrong temperature

Too-high temperature = nonsense; temperature of 0 = identical, repetitive text.

❌ Same task, different wording, different answer (prompt sensitivity)

A tiny rephrase changed the result because the model reacts to exact tokens.

Time to fade the scaffolding. Starting from the word "the", greedily generate three more tokens and print the finished sentence. The starter below gives you only a comment outline — write the logic yourself.

Lesson complete — you can now explain how an LLM works!

You learned that an LLM is a decoder-only transformer doing next-token prediction over tokens within a context window, that its knowledge lives in billions of parameters, and that temperature, softmax, pretraining vs fine-tuning, scaling laws, and hallucination shape what it produces.

🚀 Up next: Tokenization Strategies — see exactly how text becomes the tokens an LLM reads.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary