top of page

Debug Production Issues — Debugging Systematically, Part 8

Shift Quality Editorial
Jul 20
3 min read

Updated: Oct 4

Debugging Systematically · Part 8

Debugging in production is different from debugging at your desk: the clock is running, real users are affected, and the wrong move makes it worse. The instinct to dive straight into the code is usually the mistake — the first job is to stop the bleeding, not to understand everything. This walks through handling a live incident in order: mitigate first, gather observability, narrow the scope, then find the cause.


Production is fragile, observed, and the consequences of mistakes are real. Different discipline than local debugging.


Step 1: Stop the Bleeding First (5 min)


Before investigating: is the system actively harming users?


  • High error rate? Roll back.

  • Performance degraded? Scale up.

  • Specific endpoint broken? Disable via feature flag.


Stabilize first; root-cause later. Users' experience matters more than your understanding.


Step 2: Don't Make It Worse (5 min)


Production rules:


  • No experimental changes

  • No new code while debugging

  • One change at a time

  • All changes reversible


A common pattern: "let me try this fix" → makes the issue different → original cause obscured → harder to diagnose.


Discipline first; cleverness later.


Step 3: Gather Observability (10 min)


What's already instrumented:


  • Metrics dashboards (Prometheus, Datadog, etc.)

  • Logs (Kibana, Loki, etc.)

  • Traces (Jaeger, Tempo, etc.)

  • APM (Datadog, New Relic, etc.)


Look here first. The data may already explain the issue.


Step 4: Time Window (10 min)


Pin down when the issue started:


  • Error rate change in metrics

  • First "failed" log line

  • First user report

  • Last deploy / config change


14:31 — deploy completes
14:32 — error rate spike begins  ← likely cause

Correlate the start of the issue with anything that changed.


Step 5: Identify Scope (10 min)


How widespread?


  • All users? Specific cohort?

  • All regions? Specific datacenter?

  • All endpoints? Specific endpoint?

  • All requests? Some condition?


Narrowing scope narrows the cause.


Errors only on POST /api/checkout
Only for users with non-US shipping address
Started at 14:32

Three signals. Now the cause is in a small area.


Step 6: Read Recent Changes (10 min)


git log --since="2 hours ago"

Look for recent commits, especially around the affected code path.


Config changes:


# Terraform diffs, k8s diffs, env var changes

Often the cause is "Sam shipped a change at 14:30."


Step 7: Read Logs by Trace ID (10 min)


Find a failed request; trace its journey:


[abc123] gateway: POST /checkout received
[abc123] gateway: → auth service
[abc123] auth: validated
[abc123] gateway: → checkout service
[abc123] checkout: ERROR — connection refused to payment-service

Found the failure point. Now: why is payment-service unreachable?


Step 8: Minimal Reproduction (15 min)


If you can:


  • Reproduce in staging

  • Or in a sandbox

  • Or with a synthetic request


Don't experiment on production. Once reproduced, debug normally.


# Replay a failing request against staging
curl -X POST https://staging.example.com/checkout \
  -d @failed-request-body.json

Step 9: Communicate (10 min)


Keep the team informed:


  • Status updates every 15-30 minutes during outage

  • "I see X; trying Y" (not silent)

  • When resolved, full summary


Status pages, Slack channels, ticket comments. Whatever your team uses.


Customer-facing comms separately; let designated comms lead handle.


Step 10: After: Post-Mortem (10 min)


Once stable:


  • Document the timeline

  • What caused it

  • What detected it (or who)

  • How it was fixed

  • What can prevent recurrence

  • What can detect it earlier next time


Blameless. Focus on the system. (See Part 9.)


What You Just Did


Production debugging: stabilize first, gather observability, narrow scope, correlate with changes, read by trace ID, reproduce safely, communicate, post-mortem. The discipline of live debugging.


Common Failure Modes


Debug before mitigating. Users suffer while you investigate.


Experimenting in prod. Breaks more things.


Silent investigation. Team doesn't know status.


Skip post-mortem. Same issue recurs.


No observability. Debugging without data.



Continue the Debugging Systematically path



For the reasoning behind this step-by-step guide, read Debugging in Production Without Losing Your Mind.


Part of the Debugging Systematically learning path.

bottom of page