Feature Engineering & Selection

Better inputs beat fancier models. Learn to scale, encode, combine, and bin features — then select the ones that matter while avoiding the trap of data leakage.

Learn Feature Engineering & Selection in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

A great chef spends most of the time on prep — washing, chopping, and measuring — before anything hits the pan. The cooking method matters, but raw, badly prepped ingredients ruin even a perfect recipe. Feature engineering is the prep work of machine learning.

Scaling is measuring to the same units, encoding is translating words into numbers the model can taste, and feature selection is throwing out the ingredients that don't belong. Get the prep right and a simple model cooks beautifully.

Scaling puts numeric features on a comparable footing so no single large-range feature dominates. Two common choices: standardisation (mean 0, std 1) and min-max scaling (squash to 0..1). Distance- and gradient-based models need it; trees do not.

Encoding turns categories into numbers. Use one-hot encoding for unordered categories (one 0/1 column each) so you never imply a false order. Plain label encoding is fine only for genuinely ordered categories (small medium large).

Let's implement the two most common transforms: standardising a numeric feature and one-hot encoding a category. Run it and watch the outputs.

Sometimes the raw features hide the pattern. Three ways to surface it:

More features isn't always better — irrelevant ones add noise and overfitting. Three families of feature selection help you keep the useful ones:

In practice scikit-learn provides ready transforms. Study this structural reference (it isn't runnable here) and notice the golden rule: every transform is fit on the training set only , then applied to the test set.

Wrapping these steps in a Pipeline makes the fit-on-train-only rule automatic and leakage-proof.

Fill in the blanks so min_max() squashes values into the 0..1 range. The expected output is in the comments.

Group a continuous age into categories. Fill in the two blanks to return the right bucket.

Create an interaction feature and show it perfectly predicts the target. Only a comment outline is provided.

These feature-engineering mistakes silently inflate your scores. Watch for them.

Computing scaling statistics over train + test leaks test information into training.

Mapping red/green/blue to 0/1/2 invents an order the model will misuse.

❌ A feature that secretly contains the answer

Including a column that is derived from the label gives unbeatable validation scores that vanish in production.

✅ Fix: drop anything unknown at prediction time:

You can now scale and encode features correctly, craft interaction, polynomial, and binned features, choose between filter, wrapper, and embedded selection, and keep data leakage out of your pipeline.

🚀 Up next: Time-Series Forecasting — modelling data that unfolds over time.

Practice quiz

What is feature engineering?

  • Creating and transforming input features to help a model learn
  • Picking a model
  • Labelling the data
  • Tuning the learning rate

Answer: Creating and transforming input features to help a model learn. Feature engineering crafts better inputs — transforming, combining, or creating features so the model can find patterns more easily.

Which encoding turns a categorical column into one binary column per category?

  • Standard scaling
  • One-hot encoding
  • Log transform
  • Binning

Answer: One-hot encoding. One-hot encoding creates a 0/1 column for each category, which models can use without implying a false ordering.

Why is label (ordinal) encoding risky for unordered categories like colour?

  • It is too slow
  • It implies a false numeric order (e.g. red < blue < green)
  • It removes the column
  • It needs scaling

Answer: It implies a false numeric order (e.g. red < blue < green). Mapping unordered categories to 1, 2, 3 invents an order that linear and distance-based models will wrongly exploit.

What is an interaction feature?

  • A scaled feature
  • A new feature combining two others (e.g. their product) to capture joint effects
  • A removed feature
  • A binned feature

Answer: A new feature combining two others (e.g. their product) to capture joint effects. Interaction features (like price multiplied by quantity) let a model capture effects that depend on two features together.

What does binning (discretization) do?

  • Scales features to mean 0
  • Encodes text
  • Groups a continuous feature into ranges/buckets
  • Removes outliers automatically

Answer: Groups a continuous feature into ranges/buckets. Binning converts a continuous value (like age) into ranges (child/adult/senior), which can capture non-linear effects simply.

Filter, wrapper, and embedded are three families of what?

  • Scaling methods
  • Feature selection methods
  • Encoding methods
  • Loss functions

Answer: Feature selection methods. They are the three feature-selection strategies: filter (statistics), wrapper (search with a model), embedded (selection during training).

A wrapper feature-selection method (like recursive feature elimination) works by:

  • Ranking features by a quick statistic
  • Repeatedly training a model on feature subsets to see which help
  • Using L1 penalties during training
  • One-hot encoding everything

Answer: Repeatedly training a model on feature subsets to see which help. Wrapper methods search over feature subsets, training the model each time — accurate but computationally expensive.

An embedded method such as Lasso (L1) selects features by:

  • Ranking with correlation only
  • Shrinking some coefficients to exactly zero during training
  • Trying every subset
  • Binning the target

Answer: Shrinking some coefficients to exactly zero during training. Lasso's L1 penalty drives weak features' coefficients to zero as part of fitting, doing selection inside the model.

What is data leakage in feature engineering?

  • Losing rows of data
  • Information from the test set or future sneaking into the features
  • Too many features
  • Slow training

Answer: Information from the test set or future sneaking into the features. Leakage is when info you wouldn't have at prediction time (or test data) bleeds into training, giving falsely high scores.

To avoid leakage when scaling, you should fit the scaler on:

  • The whole dataset before splitting
  • The test set
  • Nothing — never scale
  • The training set only, then transform the test set

Answer: The training set only, then transform the test set. Fit the scaler (and other transforms) on training data only; applying training statistics to the test set prevents leakage.