Debug Race Conditions — Debugging Systematically, Part 7
Updated: Jul 28
Debugging Systematically · Part 7
A race condition is the bug that passes every test on your machine and fails once a week in production, because it only appears when two things happen in exactly the wrong order. That intermittence is what makes it maddening — and there's a method for it. This walks through debugging races: recognizing the shared-state patterns behind them, reproducing them on purpose, and fixing them with atomicity or idempotency instead of hope.
A bug that vanishes when you add a print statement. Almost always a race condition. This tutorial finds them.
Step 1: Recognize a Race (5 min)
Symptoms:
Works in dev; fails in prod
Fails under load; works at low traffic
"Heisenbug" — vanishes when you look at it
Test fails on CI but passes locally
Different result with retry
When timing affects behavior, suspect a race.
Step 2: Classic Patterns (10 min)
Read-modify-write:
balance = get_balance() # 100
new_balance = balance + 10 # 110
set_balance(new_balance) # writes 110
Two threads do this concurrently:
T1: read 100
T2: read 100
T1: write 110
T2: write 110 ← should be 120
Lost update.
Check-then-act:
if not exists(file):
create(file) # another process might create first
Two processes both see "not exists"; both try to create.
Initialization:
if not _config:
_config = load_config() # could run twice
Step 3: Reproduce Deterministically (10 min)
Make the race fire on demand:
# Use a barrier to force concurrent execution
import threading
barrier = threading.Barrier(2)
def worker():
barrier.wait() # both threads pause here
increment_counter() # both fire at exactly the same time
t1 = threading.Thread(target=worker)
t2 = threading.Thread(target=worker)
t1.start(); t2.start()
t1.join(); t2.join()
assert counter == 2 # may fail if race
Forces the race. Now it's reliable.
Step 4: Add Strategic Sleeps (10 min)
To enlarge the window:
def increment():
val = get()
time.sleep(0.01) # widens the race window
set(val + 1)
Bug now reproduces 99% of the time.
For tests, use this to make a flaky test fail consistently. Then fix.
Step 5: Identify the Shared State (5 min)
Races require shared mutable state. Map yours:
Global variables
Class attributes shared across instances
Database rows
Filesystem
External services
Each is a potential race site.
Step 6: Make It Atomic (10 min)
The fix:
Locks: with mutex: around critical section
Atomic operations: counter.add(1) instead of counter = counter + 1
Compare-and-swap: "set X to Y only if X is still Z"
Database transactions: isolation level handles many cases
Pick the right tool for the granularity.
import threading
counter = 0
lock = threading.Lock()
def increment():
with lock:
global counter
counter += 1
Coarse-grained but correct.
Step 7: Make It Idempotent (10 min)
For some races, instead of preventing them, make duplication safe:
def deposit(account_id, amount, idempotency_key):
if already_processed(idempotency_key):
return
with transaction():
update_balance(account_id, amount)
record_processed(idempotency_key)
Two retries with the same key = one effect.
Step 8: Linter Tools (10 min)
Some races are detectable:
Go: go test -race (race detector)
C/C++: ThreadSanitizer
Java: FindBugs, SpotBugs
Rust: the borrow checker catches many at compile time
Where available, use them in CI.
Step 9: Stress Test (10 min)
For "rare bug in prod":
# Run the test 100 times to surface flakiness
pytest --count=100 tests/test_flaky.py
Or in app code: load-test under high concurrency. Race conditions surface as the timing window widens.
Step 10: Document Concurrency (10 min)
For each piece of shared state, document:
Who reads
Who writes
What protects it
What invariants must hold
class Cache:
"""Thread-safe cache.
Reads: any thread, lock-free
Writes: synchronized via self._lock
Invariant: keys never disappear; values are immutable
"""
Future maintainers know what's safe.
What You Just Did
Race condition debugging: recognize, reproduce, identify shared state, fix with atomicity or idempotency, use linters, stress test, document. The unique skill of concurrent debugging.
Common Failure Modes
"Can't reproduce." Use barriers + sleeps to force.
Adding more locks until it works. Coarse-grained; degrades perf; sometimes wrong lock.
Trust serial test. Concurrent code needs concurrent tests.
No race detector. Catches half the bugs at compile time.
Database race assumed away. Default isolation often doesn't protect.
Continue the Debugging Systematically path
Previous — Part 6: Debug Memory Leaks
Part of the Debugging Systematically learning path.
