Natural Language Processing

Turn raw text into numbers a model can learn from — clean it, count it, weight it, and read its sentiment.

Learn Natural Language Processing in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Imagine teaching a child who has never seen language to read. You don't hand them a novel. You start by showing that text breaks into words (tokenization). You teach that "Cat" and "cat" are the same word (lowercasing) and that commas aren't words (removing punctuation). You explain that tiny words like "the" and "is" appear everywhere and rarely change the meaning (stopwords).

Then you teach which words matter in a given sentence (TF-IDF), that some words travel in pairs like "ice cream" (n-grams), and finally that words have meanings — "happy" sits near "joyful" and far from "miserable" (embeddings). A computer learns to read the exact same way, but with maths standing in for intuition. That is all NLP is: a ladder from raw characters up to meaning.

Raw text is messy: mixed case, punctuation, and filler words. Preprocessing is the cleanup that happens before any model sees your text. You lowercase it, tokenize it (split it into a list of words), drop punctuation, remove stopwords (common words like "the" and "is" that carry little meaning), and reduce word forms with stemming or lemmatization .

Stemming crudely chops endings (so running becomes runn ) and can produce non-words. Lemmatization is smarter: it uses a dictionary to return a real base word ( running becomes run ). Read the worked example, then run it to watch each step transform the text.

Models do maths, not words, so you must convert text into vectors (lists of numbers). Bag of Words (BoW) is the simplest: build a vocabulary of every unique word, then represent each document by how many times each vocab word appears. Order is thrown away — hence "bag".

The problem: common words like "the" dominate the counts without adding meaning. TF-IDF (Term Frequency × Inverse Document Frequency) fixes this. It multiplies each word's frequency in a document by how rare the word is across all documents, so distinctive words score higher and ubiquitous words score lower.

In real projects you don't hand-roll the counting — scikit-learn's CountVectorizer and TfidfVectorizer do it in two lines. Read this example and compare it to the # Expected output comments:

Fill in the two blanks to split a sentence into words and count each one. This is Bag of Words for a single document, built by hand.

Bag of Words throws away order, so "dog bites man" and "man bites dog" look identical to it. An n-gram is a sequence of n adjacent words: a bigram is a pair, a trigram is three. Including bigrams like ("natural", "language") lets your model capture phrases a single-word bag would miss — at the cost of a much larger vocabulary.

Term frequency (the "TF" in TF-IDF) is just a word's count divided by the total number of words. Fill in the blank to finish the calculation.

BoW and TF-IDF have a blind spot: to them, "happy" and "joyful" are as unrelated as "happy" and "table" — just different columns. Word embeddings fix this by representing each word as a dense vector of numbers (often 100–300 of them) learned from how words are actually used together.

Words used in similar contexts end up close together in this vector space. The famous result is that you can do arithmetic with meaning: king − man + woman ≈ queen .

word2vec — learns embeddings by predicting a word from its nearby context words (a sliding window). Fast and famous.

GloVe — learns from global co-occurrence counts (how often word pairs appear together across the whole corpus).

Both produce a lookup table mapping each word to its vector. Modern transformers (BERT, GPT) take this further with contextual embeddings — the vector for "bank" differs in "river bank" versus "bank account" — but the intuition you've just learned is the same.

Everything in this lesson clicks together into one of NLP's most common tasks: deciding whether a review is positive or negative. Here is the full pipeline, stage by stage:

Production systems swap steps 4–5 for a fine-tuned transformer like BERT, which handles context, negation, and sarcasm far better — but the shape of the pipeline is exactly this.

These four mistakes trip up almost everyone learning NLP. Spotting them early saves hours.

Feeding raw text straight into a vectorizer means "Hello," and "hello" become different tokens, bloating the vocabulary with near-duplicates.

✅ Fix: always preprocess (lowercase, strip punctuation, drop stopwords) before counting.

Treating "AI" , "ai" , and "ai!" as three separate words splits one concept across three columns and weakens every signal.

✅ Fix: call .lower() and remove punctuation so word forms collapse together.

Adding bigrams and trigrams to a large corpus can create hundreds of thousands of columns — slow to train and prone to overfitting.

✅ Fix: cap the vocabulary with CountVectorizer(max_features=5000) or min_df to drop rare terms.

Calling fit_transform on the whole dataset before splitting lets the vectorizer learn the test vocabulary — your accuracy looks great, then collapses in production.

✅ Fix: split first, fit only on train, transform test:

Time to fly with less support. The starter below gives you only a comment outline — no filled-in logic. Use everything from this lesson (tokenize, then count matches against two word sets) to print a verdict.

Lesson 9 complete — you can teach a computer to read!

You can preprocess raw text, build Bag of Words and TF-IDF vectors by hand and with scikit-learn, generate n-grams, explain word embeddings, and lay out a full sentiment-analysis pipeline — including how to avoid data leakage when fitting a vectorizer.

🚀 Up next: Computer Vision — teaching computers to see and understand images.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary