Concurrency
Master Python's concurrency models and learn when to use threads, processes, or AsyncIO for maximum performance.
Learn Python, JavaScript, Java and more with free interactive lessons, real projects and AI-powered help. Beginner-friendly.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
What You'll Learn in This Lesson
A thread is a lightweight unit of execution within a single process.
The GIL (Global Interpreter Lock) is a mutex that protects access to Python objects.
(e.g., image processing, hashing, compression)
A process is a full Python interpreter with its own memory.
Threads shine because requests are I/O-bound.
This is how production-grade scrapers, ML preprocessors, and automation bots work.
Python uses cooperative + preemptive scheduling for threads.
The operating system decides when each thread runs, based on CPU availability.
Inside Python, only ONE thread can execute Python bytecode at once.
Let's say you need to download 10,000 images.
Sequential time = 10,000 × (0.4 seconds each) = ~4000 seconds (1.1 hours)
Threaded time (200 threads) = ~20 seconds total
Because threads wait most of the time, so Python overlaps waits.
If you run these in threads → NO speed improvement .
If you run these in processes → 4× faster on 4 cores, 12× on 12 cores, etc.
The next sections cover professional-level concurrency patterns:
Lock, RLock, Event, Semaphore, Queue — Safe shared memory primitives
multiprocessing.Queue, Pipe, Manager, shared memory arrays
14. ThreadPoolExecutor vs ProcessPoolExecutor
concurrent.futures abstraction for both models
Combining AsyncIO + Threads + Processes like YouTube/Instagram
Race conditions, deadlocks, blocking calls, serialization issues
CPU-heavy: 8× with processes; I/O-heavy: 50× with threads
✔ Threads for I/O | ✔ Processes for CPU | ✔ AsyncIO for massive concurrency
counter += 1 is NOT atomic. It's 3 instructions:
Locks ensure only ONE thread accesses critical code at once.
But… ❗ Locks introduce blocking, which could slow threads.
When threads freeze forever waiting on each other
What breaks when passing objects to processes
29. Real Architecture: High-Performance Scraper
ProcessPool for CPU, ThreadPool for I/O, AsyncIO for APIs
Python 3.13+ will enable true parallel threads for CPU work
You're operating at professional backend engineer level now.
📋 Quick Reference — Concurrency
You now understand the GIL, when to use threads vs processes, and how to safely share data between concurrent workers.
Up next: Parallelism — use concurrent.futures for a clean high-level API over threads and processes.
Practice quiz
What is the difference between concurrency and parallelism?
- They are the same thing
- Parallelism is slower than concurrency
- Concurrency switches between tasks; parallelism does multiple things at the exact same time
- Concurrency requires multiple CPU cores
Answer: Concurrency switches between tasks; parallelism does multiple things at the exact same time. Concurrency interleaves tasks; parallelism runs them truly simultaneously on multiple cores.
What is the GIL (Global Interpreter Lock)?
- A mutex that lets only one thread run Python bytecode at a time
- A networking protocol
- A way to lock files
- A garbage collector
Answer: A mutex that lets only one thread run Python bytecode at a time. The GIL is a mutex ensuring only one thread executes Python bytecode at a time.
For CPU-bound work, which approach actually achieves true parallelism in Python?
- threading
- asyncio
- None — Python can't parallelize
- multiprocessing
Answer: multiprocessing. Each process has its own interpreter and GIL, so multiprocessing gives true CPU parallelism.
Why do threads work well for I/O-bound tasks despite the GIL?
- The GIL is disabled for threads
- A thread releases the GIL while waiting on I/O, letting another thread run
- I/O tasks don't use the GIL at all
- Threads create separate interpreters
Answer: A thread releases the GIL while waiting on I/O, letting another thread run. When a thread waits on I/O it releases the GIL, so other threads can make progress.
Do threads or processes share memory by default?
- Threads share memory; processes have separate memory
- Processes share memory; threads do not
- Both share memory
- Neither shares memory
Answer: Threads share memory; processes have separate memory. Threads share the same memory; processes are isolated and need serialization to share data.
Why is counter += 1 unsafe across multiple threads without a lock?
- It is always atomic and safe
- Integers can't be shared
- It is not atomic — load, add, and store can interleave
- The GIL prevents all sharing
Answer: It is not atomic — load, add, and store can interleave. counter += 1 is three steps (load, add, store); interleaving threads corrupt the value — a race condition.
What is the recommended way to use a threading.Lock around shared data?
- lock.acquire() and forget to release
- with lock:
- Set lock = True
- No lock is needed
Answer: with lock:. with lock: auto-acquires and auto-releases, which is the safest pattern.
Compared with threads, how do processes generally start up?
- Faster, in microseconds
- Instantly with zero cost
- At the same speed as threads
- Slower, in milliseconds (a new interpreter must spawn)
Answer: Slower, in milliseconds (a new interpreter must spawn). Processes are slower to start since each spawns its own Python interpreter and memory.
Which model is best for handling massive numbers of lightweight network tasks?
- multiprocessing
- asyncio
- One thread per task
- Pure sequential code
Answer: asyncio. asyncio scales to massive lightweight, network-first concurrency on a single thread.
Running heavy number-crunching in 100 threads gives what result in CPython?
- A ~100x speedup
- A guaranteed crash
- No real speedup — the GIL serializes CPU-bound bytecode
- True parallelism across cores
Answer: No real speedup — the GIL serializes CPU-bound bytecode. The GIL serializes Python bytecode, so CPU-bound threads get no real speedup — use processes.