Advanced NLP: BERT, T5 & LLaMA

By the end you'll explain how transformers read context, tell BERT (understanding) apart from GPT (generation), fine-tune a pretrained model, and run NER, question answering, and summarization with a single line of Hugging Face code.

Learn Advanced NLP: BERT, T5 & LLaMA in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Imagine two readers handed the sentence "I went to the bank." A dictionary-only reader sees the word "bank" and shrugs — it could be money or a river. A thoughtful reader looks at the words around it ("I fished off the …") and instantly knows which one you mean.

That thoughtful reader is a transformer. It doesn't store one fixed meaning per word — it builds a contextual embedding , a fresh vector for each word shaped by its neighbours. That is the whole reason modern NLP understands meaning , not just spelling. BERT is the detective reading the full report in both directions to understand; GPT is the storyteller writing one word at a time; T5 is the translator who reads everything, then writes a fresh version.

An embedding is a list of numbers (a vector) that represents a piece of text so a computer can do maths with meaning. The breakthrough of transformers is that the embedding is contextual : the same word gets a different vector depending on the sentence.

You compare two vectors with cosine similarity — a score from 0 (unrelated) to 1 (same direction). This single tool powers semantic search : find the document whose vector points the same way as your question, even if it shares no exact words. Run the worked example below and read the comments — every result is stated inline.

BERT is an encoder . It is pretrained with masked language modeling (MLM) : random words are hidden behind a [MASK] token and BERT must guess them using context from both sides. Because it reads in both directions, it is brilliant at understanding — classification, NER, similarity, and question answering.

GPT is a decoder . It is pretrained to predict the next token using only the words to its left. That left-to-right view makes it a natural generator — chat, writing, code. T5 combines both: an encoder reads the input, a decoder writes the output, which is the perfect shape for summarization and translation.

You almost never train an NLP model from zero. Instead you use transfer learning : take a model already pretrained on billions of words, then fine-tune it on your much smaller, task-specific dataset. The model already "knows" language; you only teach it your specific job.

The recipe for fine-tuning BERT on a task like NER or sentiment is short:

With a few thousand labelled examples you can reach accuracy that would have needed millions of examples to train from scratch. That is the power — and the economy — of transfer learning.

Three of the most common NLP jobs each have a ready-made pretrained model:

Hugging Face's pipeline() downloads and wires up the right model for you. Study the worked example below — each call is followed by its # Expected output .

Real NER uses BERT, but the idea is simple enough to do by hand: look each word up in a dictionary of known entities and label it. Fill in the one blank, then run it and check your output against the comment.

This is how a search box "understands" a question: turn the query and each document into vectors, then rank documents by cosine similarity. Complete the cosine formula and confirm the password document scores higher than the weather one.

Time to fade the scaffolding. Only a comment outline is provided — you write the logic. Build a tiny lexicon-based sentiment tagger that decides whether a review is positive, negative, or mixed.

Treating a word as one fixed meaning (old word2vec thinking). "Apple" the company and "apple" the fruit are different — a model that ignores context will confuse them.

✅ Fix: Use a transformer model whose embeddings are contextual; never average away the sentence.

Using GPT for plain classification or NER. It works, but it is 10–100× slower and pricier than a fine-tuned BERT, and often less accurate on the label.

✅ Fix: Encoder (BERT) for understanding, decoder (GPT) for generation, encoder-decoder (T5) for rewriting. Use the simplest model that solves the task.

BERT caps at 512 tokens. Feed a long document straight in and it is silently truncated — the model never sees the end, so your answer or summary is wrong.

✅ Fix: Chunk long text into overlapping windows, or use a long-context model designed for it.

Fine-tuning on noisy, mislabelled, or imbalanced data. The model faithfully learns your mistakes — garbage in, garbage out — and scores look fine on a flawed test set.

✅ Fix: Clean and balance labels, hold out a trustworthy validation set, and prefer fewer correct examples over many noisy ones.

Need to rewrite text (summarize/translate)? Use an encoder-decoder like T5 or BART — it has both.

Lesson 35 complete — you understand the NLP model landscape!

You can explain contextual embeddings, distinguish BERT's encoder from GPT's decoder, describe transfer learning and fine-tuning, and run NER, question answering, and summarization with Hugging Face's pipeline() . You also built a rule-based NER tagger and a cosine-similarity search by hand.

🚀 Up next: RAG Systems — combine these LLMs with a knowledge base so they answer from your documents, not just their training data.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary