Fine-Tuning LLMs: LoRA & QLoRA

By the end of this lesson you'll know when to prompt, when to use RAG, and when to fine-tune — and how to prepare data and train an adapter without melting your GPU.

Learn Fine-Tuning LLMs: LoRA & QLoRA in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

A pre-trained LLM is like a sharp new graduate who knows a bit about everything. You have three ways to get the work you need from them, in increasing cost:

LoRA is the smart twist: instead of re-educating the whole employee, you teach them one focused playbook (a tiny adapter) they slot in for your task — and can pop out again for the next client.

Fine-tuning is powerful, but it is the last tool you should reach for, not the first. Most problems are solved more cheaply by writing a better prompt or by retrieval. Use this rule of thumb:

The runnable example below builds a tiny decision helper and shows how few parameters LoRA actually trains. Run it and read every comment.

Full fine-tuning updates every weight in the model. It gives the best quality but is brutally expensive: training a 70B model in 32-bit needs hundreds of gigabytes of GPU memory just to hold the weights, gradients, and optimiser state.

PEFT (Parameter-Efficient Fine-Tuning) is the family of methods that avoid this. The most popular is LoRA (Low-Rank Adaptation): freeze the original weights and learn two small matrices whose product is added to a weight matrix. A full weight matrix holds d×d numbers; LoRA trains only 2×d×r , where the rank r (usually 8–16) is tiny. That is often under 1% of the parameters.

QLoRA stacks one more trick on top: load the frozen base model in 4-bit precision (NormalFloat4) instead of 16-bit. The adapters stay in higher precision and do the learning, while the giant frozen base barely takes up room — so a 70B model that needed four A100s can be fine-tuned on a single 48GB GPU.

Here is what a real LoRA setup looks like with Hugging Face peft and transformers . It's read-only (it needs a GPU and the libraries), but notice the trainable% line in the expected output:

And QLoRA is the same idea with one extra config object to quantise the base model to 4-bit:

Instruction tuning is just supervised fine-tuning on instruction → response pairs. It teaches the model to follow commands rather than only continue text. This is the workhorse you'll use most.

RLHF (Reinforcement Learning from Human Feedback) goes further to make answers preferred by people. Humans rank several model answers; a small reward model learns to predict those rankings; then the LLM is optimised to score highly under that reward model. It works well but has many moving parts.

DPO (Direct Preference Optimisation) reaches a similar place far more simply. You give it pairs of a chosen answer and a rejected answer, and it trains the model directly to prefer the chosen one — no separate reward model, no reinforcement-learning loop. For most teams today, DPO is the easier path to alignment.

A fine-tune is only as good as its data. You build a list of records, each with a prompt (the instruction, wrapped in a consistent template) and a completion (the ideal response). The template matters: the model learns to produce whatever follows your ### Response: marker, so it must be identical in training and at inference.

Good data is consistent (same format every row), clean (no empty or contradictory answers), and representative (covers the cases you'll actually see). The runnable example below formats raw pairs into records and then shows the core idea of training — a single weight learning by gradient descent. Run it:

That five-line loop is the whole secret of training, scaled up: predict, measure the error, nudge each weight a little against the error, repeat. Fine-tuning runs this over billions of weights (or, with LoRA, just the adapter's few million).

Always hold back a validation set the model never trains on. During training you watch two numbers: training loss (error on data it sees) and validation loss (error on data it doesn't). When training loss keeps falling but validation loss starts rising , you are overfitting — the model is memorising your examples instead of learning the pattern.

Finish the to_record function so each FAQ pair becomes a clean instruction/response record. Fill in the blanks marked ___ and check your output against the expected result in the comments.

Complete the choose function so it returns the right approach for each task. Fill in the blanks marked ___ .

No blanks this time — just a brief and an outline. Drop the bad records and format the rest. Use the expected output to check yourself.

You spin up GPUs and label data, but a one-line prompt tweak already solved it.

✅ Fix: Always try prompting first, then RAG. Only fine-tune when a careful prompt still can't make the behaviour reliable.

A dozen examples can't teach a behaviour; the model barely shifts or learns the noise.

✅ Fix: Aim for a few hundred to a few thousand clean, consistent examples. Quality and consistency beat raw volume.

After training, the model nails your task but is suddenly worse at general questions.

✅ Fix: Use a small learning rate, fewer epochs, and LoRA (which leaves the base weights frozen). Mix in some general examples if needed.

Training loss keeps dropping while validation loss climbs — it's memorising, not learning.

✅ Fix: Stop early when validation loss turns up, hold out a validation set, and train for only 1–3 epochs.

Great validation scores, terrible real answers — because inference uses a different template than training.

✅ Fix: Keep the prompt template (markers, whitespace, newlines) byte-for-byte identical at train and inference time.

Lesson complete — you can now reason about fine-tuning like a practitioner!

You can pick prompting, RAG, or fine-tuning for a task; explain LoRA, QLoRA, and PEFT; describe instruction tuning, RLHF, and DPO; prepare clean instruction/response data; and guard against overfitting, catastrophic forgetting, and format mismatch.

🚀 Up next: Reinforcement Learning Basics — the reward-driven training that powers RLHF and decision-making agents.

Practice quiz

What is fine-tuning a large language model?

  • Training a model from scratch on the whole internet
  • Writing a longer prompt
  • Continuing training of a pretrained model on your own examples to adapt its behaviour
  • Compressing the model to fewer bits

Answer: Continuing training of a pretrained model on your own examples to adapt its behaviour. Fine-tuning starts from a capable base model and nudges its weights with a small, focused dataset.

When should you prefer RAG over fine-tuning?

  • When the model needs private or changing FACTS
  • When you need a new consistent behaviour or style
  • When you have no GPU at all
  • When you want to reduce model size

Answer: When the model needs private or changing FACTS. RAG injects documents at query time, so it adds knowledge that can change without retraining. Fine-tuning changes behaviour.

What does LoRA (Low-Rank Adaptation) do?

  • Updates every weight in the model
  • Quantises the model to 4-bit
  • Removes attention layers
  • Freezes the base weights and trains two small low-rank adapter matrices

Answer: Freezes the base weights and trains two small low-rank adapter matrices. LoRA freezes the base model and learns small low-rank matrices, often under 1% of the parameters.

What extra trick does QLoRA add on top of LoRA?

  • It trains all weights in full precision
  • It loads the frozen base model in 4-bit precision to save memory
  • It removes the adapter matrices
  • It uses a larger learning rate

Answer: It loads the frozen base model in 4-bit precision to save memory. QLoRA loads the frozen base in 4-bit (NormalFloat4), so even a 70B model can fine-tune on a single GPU.

What does PEFT stand for?

  • Parameter-Efficient Fine-Tuning
  • Pretrained Embedding Fine-Tuning
  • Partial Encoder Feature Transfer
  • Precision Enhanced Forward Training

Answer: Parameter-Efficient Fine-Tuning. PEFT is the family of Parameter-Efficient Fine-Tuning methods, of which LoRA is the most popular.

What is instruction tuning?

  • Reinforcement learning with a reward model
  • Quantising the model to int8
  • Supervised fine-tuning on instruction/response pairs so the model follows commands
  • Retrieving documents at query time

Answer: Supervised fine-tuning on instruction/response pairs so the model follows commands. Instruction tuning is supervised fine-tuning on instruction-response pairs, teaching the model to follow commands.

How does DPO differ from RLHF?

  • DPO needs a separate reward model and an RL loop
  • DPO trains directly on chosen vs rejected answer pairs, with no separate reward model
  • DPO does not change the model weights
  • DPO only works for image models

Answer: DPO trains directly on chosen vs rejected answer pairs, with no separate reward model. DPO (Direct Preference Optimisation) trains directly on preferred/rejected pairs, avoiding RLHF's reward model and RL loop.

How do you detect overfitting during fine-tuning?

  • Training loss rises while validation loss falls
  • Both losses fall together forever
  • The model size increases
  • Training loss keeps falling while validation loss starts rising

Answer: Training loss keeps falling while validation loss starts rising. When training loss drops but validation loss climbs, the model is memorising the training data rather than generalising.

What is catastrophic forgetting?

  • The model crashes during training
  • The model gets great at the new task but worse at general skills
  • The optimiser loses the gradients
  • The dataset is deleted after training

Answer: The model gets great at the new task but worse at general skills. Training too hard or on too narrow data can make the model lose its previously learned general abilities.

Why must the prompt template match between training and inference?

  • To save disk space
  • To reduce the number of parameters
  • Because the model learns to respond to the exact cue it saw in training
  • Because libraries require it

Answer: Because the model learns to respond to the exact cue it saw in training. If training uses '### Response:' but inference uses a different marker, the model never sees its learned cue and quality collapses.