top of page

Debug Production Issues — Debugging Systematically, Part 8

  • Contributor
  • 1 day ago
  • 3 min read
Debugging Systematically · Part 8

Production is fragile, observed, and the consequences of mistakes are real. Different discipline than local debugging.

Step 1: Stop the Bleeding First (5 min)

Before investigating: is the system actively harming users?

  • High error rate? Roll back.

  • Performance degraded? Scale up.

  • Specific endpoint broken? Disable via feature flag.

Stabilize first; root-cause later. Users' experience matters more than your understanding.

Step 2: Don't Make It Worse (5 min)

Production rules:

  • No experimental changes

  • No new code while debugging

  • One change at a time

  • All changes reversible

A common pattern: "let me try this fix" → makes the issue different → original cause obscured → harder to diagnose.

Discipline first; cleverness later.

Step 3: Gather Observability (10 min)

What's already instrumented:

  • Metrics dashboards (Prometheus, Datadog, etc.)

  • Logs (Kibana, Loki, etc.)

  • Traces (Jaeger, Tempo, etc.)

  • APM (Datadog, New Relic, etc.)

Look here first. The data may already explain the issue.

Step 4: Time Window (10 min)

Pin down when the issue started:

  • Error rate change in metrics

  • First "failed" log line

  • First user report

  • Last deploy / config change

14:31 — deploy completes
14:32 — error rate spike begins  ← likely cause

Correlate the start of the issue with anything that changed.

Step 5: Identify Scope (10 min)

How widespread?

  • All users? Specific cohort?

  • All regions? Specific datacenter?

  • All endpoints? Specific endpoint?

  • All requests? Some condition?

Narrowing scope narrows the cause.

Errors only on POST /api/checkout
Only for users with non-US shipping address
Started at 14:32

Three signals. Now the cause is in a small area.

Step 6: Read Recent Changes (10 min)

git log --since="2 hours ago"

Look for recent commits, especially around the affected code path.

Config changes:

# Terraform diffs, k8s diffs, env var changes

Often the cause is "Sam shipped a change at 14:30."

Step 7: Read Logs by Trace ID (10 min)

Find a failed request; trace its journey:

[abc123] gateway: POST /checkout received
[abc123] gateway: → auth service
[abc123] auth: validated
[abc123] gateway: → checkout service
[abc123] checkout: ERROR — connection refused to payment-service

Found the failure point. Now: why is payment-service unreachable?

Step 8: Minimal Reproduction (15 min)

If you can:

  • Reproduce in staging

  • Or in a sandbox

  • Or with a synthetic request

Don't experiment on production. Once reproduced, debug normally.

# Replay a failing request against staging
curl -X POST https://staging.example.com/checkout \
  -d @failed-request-body.json

Step 9: Communicate (10 min)

Keep the team informed:

  • Status updates every 15-30 minutes during outage

  • "I see X; trying Y" (not silent)

  • When resolved, full summary

Status pages, Slack channels, ticket comments. Whatever your team uses.

Customer-facing comms separately; let designated comms lead handle.

Step 10: After: Post-Mortem (10 min)

Once stable:

  • Document the timeline

  • What caused it

  • What detected it (or who)

  • How it was fixed

  • What can prevent recurrence

  • What can detect it earlier next time

Blameless. Focus on the system. (See Part 9.)

What You Just Did

Production debugging: stabilize first, gather observability, narrow scope, correlate with changes, read by trace ID, reproduce safely, communicate, post-mortem. The discipline of live debugging.

Common Failure Modes

Debug before mitigating. Users suffer while you investigate.

Experimenting in prod. Breaks more things.

Silent investigation. Team doesn't know status.

Skip post-mortem. Same issue recurs.

No observability. Debugging without data.

Continue the Debugging Systematically path

Part of the Debugging Systematically learning path.

bottom of page