top of page

Debug Distributed Systems — Debugging Systematically, Part 5

Shawn West
Jul 15
3 min read

Updated: Jul 28

Debugging Systematically · Part 5

Debugging one service is hard; debugging a request that hops through six of them, each with its own logs, is a different problem entirely — the evidence is scattered and the bug is often in the space between services. This walks through making a distributed system debuggable: trace IDs to stitch a single request back together, correlating logs across services, and finding the failures that live at the edges.

A request hits 5 services. One fails. Which one? Debuggers don't work across services. You need other tools.

Step 1: Trace IDs Are the Glue (10 min)

Generate a trace ID at the edge; propagate to every service:

Request → Gateway: trace=abc123
Gateway → Auth: trace=abc123
Gateway → Orders: trace=abc123
Orders → Payment: trace=abc123

Logs and traces include the ID:

[abc123] gateway: received POST /order
[abc123] auth: validated user 42
[abc123] orders: created order 7
[abc123] payment: charging $10
[abc123] payment: ERROR — gateway timeout

Search all services for abc123; see the whole story.

W3C Trace Context is the standard. OpenTelemetry handles propagation.

Step 2: Centralized Logging (10 min)

Logs in many services = useless without aggregation.

Tools:

  • Elasticsearch + Kibana / OpenSearch: classic

  • Loki: lightweight; from Grafana

  • Datadog Logs / New Relic: managed

  • CloudWatch Logs: AWS-native

Index by service, env, time, trace ID. Search across all services with one query.

Step 3: Distributed Tracing (15 min)

Beyond logs: traces show the timing of each step.

trace abc123 (250ms total)
├── gateway: 250ms
│   ├── auth: 20ms
│   ├── orders: 220ms     ← suspicious
│   │   ├── db query: 5ms
│   │   ├── payment: 200ms ← here
│   │   │   └── stripe API: 195ms

Find the slow span. That's where to focus.

OpenTelemetry → Jaeger / Tempo / Honeycomb / Datadog APM.

Step 4: Correlate Across Services (10 min)

A trace alone tells you "this slow." Logs tell you "this error."

Use the trace ID to correlate:

  1. Find the slow trace

  2. Filter logs by that trace ID

  3. See the application context: "tried 3 retries", "fell back", "timed out"

Two views; same data; richer picture.

Step 5: Compare Healthy vs. Broken (10 min)

For "intermittent" issues:

  • Find a successful trace; baseline behavior

  • Find a failed trace; diff

What's different? Different downstream service called? Different latency on one hop? Different log lines?

Comparing pinpoints anomalies.

Step 6: Look at the Edges (10 min)

In a microservice setup, network is the most common failure mode:

  • Connection refused

  • Timeout

  • DNS failure

  • TLS errors

Check network metrics:

  • HTTP error rate per service

  • p99 latency per service

  • Retry count

  • DNS resolution time

The bug is often network, not application.

Step 7: Replay Logs (5 min)

If you have full request logs:

# Replay the failing request
curl -X POST https://api.example.com/order \
  -H "Idempotency-Key: original-key" \
  -d @captured-body.json

If it reproduces in dev: bug is in the system. If it doesn't: state-dependent issue (DB content, time, etc.).

Step 8: Time and Time Zones (10 min)

Distributed systems + different time zones = chaos.

Always log timestamps in UTC ISO 8601:

2027-02-04T14:32:01.234Z

Correlating log lines across services with mixed time zones is a special hell. Stick to UTC everywhere.

Step 9: Data Consistency Issues (10 min)

"The user clicked save; the order was created; but inventory wasn't updated."

Check:

  • Did the order-service event fire?

  • Did the inventory-service receive it?

  • Did processing succeed but not commit?

  • Did a retry overwrite?

Distributed transactions are a class of bug. Use traces + event logs to walk the data through every step.

Step 10: Reproduce What You Can (10 min)

For multi-service bugs, replicate locally:

  • Docker Compose with all services

  • Synthetic traffic generator

  • Same data as production (anonymized)

Hard to set up. Pays off when "production is broken; can I poke at it" becomes "let me reproduce on my laptop."

What You Just Did

Distributed debugging: trace IDs, centralized logs, tracing, correlation, healthy-vs-broken comparison, network metrics, replay, time, consistency, reproduction. The skill for service-oriented architectures.

Common Failure Modes

No trace propagation. Cross-service correlation impossible.

Logs in many places. Have to grep 10 servers.

No tracing. Can't see timing across services.

Mixed time zones. Can't correlate events.

Treat one service in isolation. Miss the cross-service interaction that's the actual bug.

Continue the Debugging Systematically path

Part of the Debugging Systematically learning path.

bottom of page