Parallel Programming

By the end of this lesson you'll take a CPU-bound loop that crawls on one core and spread it across all your cores with Parallel.For , Parallel.ForEach , and PLINQ's .AsParallel() — and you'll do it safely, because you'll know exactly how to stop two threads corrupting the same variable. This is how you turn an 8-core machine into an 8x speed-up instead of a pile of subtle bugs.

Learn Python, JavaScript, Java and more with free interactive lessons, real projects and AI-powered help. Beginner-friendly.

Part of the free C# course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Picture a supermarket with one long queue of shoppers. Sequential code is a single checkout till: every shopper waits for the one in front, and the queue only moves as fast as that one cashier. Parallel code is opening more tills — eight cashiers serving the same queue at once, so the line clears roughly eight times faster. Parallel.For and PLINQ are the store manager shouting "open more tills!" — they automatically split the shoppers across however many cashiers (CPU cores) you have. But watch the shared till roll: if two cashiers reach for the same cash drawer at the same moment, the total goes wrong. That collision is a data race , and most of this lesson is about giving each cashier the right kind of drawer.

These two ideas get confused constantly, so pin them down now. They solve different problems and you pick by asking one question: is my program busy, or is it waiting?

The golden rule: parallelise CPU work, await I/O work. Running Parallel.ForEach over a list of HTTP calls is a classic mistake — it ties up thread-pool threads doing nothing but waiting, which is exactly what async was built to avoid.

Reach for the Parallel /PLINQ rows when the CPU is busy; reach for the last row when the program is just waiting.

1. Parallel.For, Parallel.ForEach & Parallel.Invoke

Parallel.For takes a start and an exclusive end index and runs the loop body across thread-pool threads — each iteration may land on a different core, in any order . Parallel.ForEach does the same over a collection, and Parallel.Invoke fires off several different methods at once. Because iterations run simultaneously, you must never assume an order: in the worked example below every iteration writes to its own array slot (safe), and the printed lines come out scrambled. Read it, run it twice, and notice the order changes but the work always completes.

Your turn. The program below ships and invoices six orders in parallel. Fill in the three ___ blanks to call Parallel.ForEach and Parallel.For correctly. The printed order will be different every run — that's expected — so you check the count at the end, not the order.

2. PLINQ — Parallel LINQ

Already know LINQ? Then you already know PLINQ. Drop .AsParallel() into any query and the engine partitions the data across cores and runs your .Where / .Select / .Sum on each chunk. Order-independent reductions like .Count() and .Sum() are the sweet spot — the answer is identical to the sequential version, just faster. The catch: by default the order of streamed results is arbitrary, so add .AsOrdered() when sequence matters.

Now you write a PLINQ pipeline. Take 1 to 100, keep the even numbers, square them, and sum the result — all in parallel. Because Sum doesn't care about order, the total is the same on every run. Fill in the four ___ blanks:

3. Thread Safety: Races, Interlocked, lock & Concurrent Collections

Here's where parallel code bites. The instant two threads write to the same variable, you have a data race . The classic example is count++ : it's secretly three steps — read, add one, write back — so two threads can read the same value and both write back the same result, silently losing increments. The worked example below shows the broken version (the total comes out too low), then three fixes: Interlocked for atomic counters, lock for multi-step updates, and ConcurrentDictionary for a thread-safe map.

4. Fast Parallel Sums with Thread-Local State

Locking on every single iteration is correct but slow — the threads spend their time queuing for the lock instead of computing. The professional pattern uses the four-lambda Parallel.For overload: each thread keeps a private running subtotal (zero contention), and only when a thread finishes does it merge its subtotal into the shared grand total once, atomically. Notice the key property — the threads finish in random order , but the final total is exactly the same every run . Determinism of the result, non-determinism of the schedule.

5. Exceptions: AggregateException

When a parallel body throws, the failure can't just bubble up one stack — several threads might fail at once. So the Parallel methods and PLINQ collect every error into a single AggregateException . You catch that one type, then loop over its .InnerExceptions to see each underlying problem. (Remaining iterations may still run, so don't assume the loop stops dead at the first throw.)

Parallelism is not free. Splitting the work, scheduling threads, and merging results all cost time and memory. If the per-item work is tiny or the collection is small, that overhead can make the parallel version slower than the simple loop. Don't guess — measure with a Stopwatch before and after.

Here's a realistic CPU-bound job: process 200 "images", each needing heavy per-item work. It uses Parallel.ForEach capped to Environment.ProcessorCount cores, collects results in a thread-safe ConcurrentBag , and reports order-independent stats. You understand every line now — the parallelism, the core cap, the thread-safe collection, and why the totals are deterministic.

The images finish in a random order, but Count , Sum and Max are order-independent — so the reported numbers are identical on every run.

Q: What's the difference between parallel and async?

Parallel uses many threads to do CPU work faster (more cooks). Async uses one thread efficiently while it waits on I/O (one cook not standing idle). Parallelise computation; await network/disk/database. Mixing them up — e.g. Parallel.ForEach over HTTP calls — wastes threads.

Q: Why is my output in a different order every time I run it?

Because parallel iterations run on different threads with no guaranteed schedule. That's normal and expected. If order matters, use PLINQ's .AsOrdered() , or collect into a list and sort it afterwards. For totals and counts, order doesn't matter at all.

Q: My parallel sum gives a different (too-low) total each run — why?

That's a data race. A plain total += x across threads loses updates because += isn't atomic. Use Interlocked.Add(ref total, x) , or the thread-local Parallel.For overload that sums private subtotals and merges once. The fixed version is deterministic — same total every run.

No. Splitting work and scheduling threads has overhead, so for small or cheap workloads the simple loop wins. Parallelism pays off when the data is large and each item does real CPU work. Always measure both with a Stopwatch before committing.

No blanks this time — just a brief and an outline. Sum every number from 1 to 1,000,000 using Parallel.For , but do it safely : a plain total += i is a data race, so make the add atomic with Interlocked.Add (or use the thread-local subtotal overload for extra credit). The whole point: even though the threads run in a random order, the total is identical on every run . Run it and check against the expected line in the comments.

Practice quiz

When should you reach for Parallel.For / PLINQ rather than async/await?

  • For I/O-bound waiting
  • For network calls
  • For CPU-bound work that keeps cores busy
  • For disk reads

Answer: For CPU-bound work that keeps cores busy. Parallelise CPU-bound work to keep every core busy; use async/await for I/O-bound waiting.

Is the end index of Parallel.For(0, n, ...) inclusive or exclusive?

  • Exclusive
  • Inclusive
  • It depends on the body
  • It loops forever

Answer: Exclusive. Parallel.For's end index is exclusive, just like a standard for loop using i < n.

What turns an ordinary LINQ query into a parallel (PLINQ) one?

  • .ToList()
  • .Where()
  • .Select()
  • .AsParallel()

Answer: .AsParallel(). Dropping .AsParallel() into a query partitions the data across cores.

By default, what is the order of streamed PLINQ results?

  • Always the original order
  • Arbitrary unless you add .AsOrdered()
  • Reverse order
  • Sorted ascending

Answer: Arbitrary unless you add .AsOrdered(). PLINQ returns results in arbitrary order by default; add .AsOrdered() to restore the original sequence.

Why does 'count++' across threads silently lose increments?

  • ++ is a read-modify-write, so two threads can read the same value and both write back the same result
  • ++ is atomic
  • The compiler removes it
  • It overflows

Answer: ++ is a read-modify-write, so two threads can read the same value and both write back the same result. ++ is read-modify-write and not atomic, so concurrent threads can clobber each other's updates — a data race.

Which is the lock-free way to safely increment a shared counter?

  • count++
  • lock (count)
  • Interlocked.Increment(ref count)
  • count = count + 1

Answer: Interlocked.Increment(ref count). Interlocked.Increment performs a single atomic, lock-free increment.

When a body inside Parallel.For throws, how do the failures surface?

  • As the original exception type
  • Bundled into one AggregateException with .InnerExceptions
  • They are swallowed
  • The program crashes instantly

Answer: Bundled into one AggregateException with .InnerExceptions. Failures from all threads are bundled into a single AggregateException; inspect .InnerExceptions for each error.

What does MaxDegreeOfParallelism control?

  • The loop's end index
  • The result order
  • The exception type
  • How many threads run at once

Answer: How many threads run at once. MaxDegreeOfParallelism caps how many threads run concurrently — match it to Environment.ProcessorCount for CPU work.

Why is the four-lambda Parallel.For overload faster for summing?

  • It skips iterations
  • Each thread keeps a private subtotal and merges once, avoiding per-iteration contention
  • It uses no threads
  • It runs sequentially

Answer: Each thread keeps a private subtotal and merges once, avoiding per-iteration contention. Thread-local subtotals have no contention; each thread merges its subtotal into the grand total just once, atomically.

Running Parallel.ForEach over a list of HTTP calls is a mistake because...

  • HTTP is too fast
  • It always throws
  • It ties up thread-pool threads doing nothing but waiting — use async instead
  • It needs a lock

Answer: It ties up thread-pool threads doing nothing but waiting — use async instead. I/O-bound work should use async/await + Task.WhenAll; Parallel blocks thread-pool threads that are merely waiting.