Build Debugging Tools Into Code — Debugging Systematically, Part 10
Updated: Jul 28
Debugging Systematically · Part 10
The best time to make a bug easy to debug is before it happens — while you're writing the code, not at 3 a.m. while it's on fire. Code that's built to be observed (structured logs, metrics, traces, a kill switch) turns a multi-hour production mystery into a five-minute lookup. This walks through building those debugging affordances in from the start, so future-you can see what the system is actually doing.
Future-you needs to debug this code at 3am. Help future-you. Build observability and control into the code from the start.
Step 1: Structured Logging (10 min)
Don't:
logger.info(f"User {user_id} did {action}")
Do:
logger.info("user_action", extra={"user_id": user_id, "action": action})
JSON output:
{"event": "user_action", "user_id": 42, "action": "login", "timestamp": "..."}
Searchable, filterable, aggregatable. Plain text logs lose at scale.
Step 2: Useful Log Levels (5 min)
DEBUG: verbose; off in production
INFO: notable events (request received, job completed)
WARN: something unexpected but handled (retried, fell back)
ERROR: something failed
CRITICAL: the world is on fire
Log levels let you control verbosity without code changes.
Step 3: Log Context (10 min)
Add per-request context:
# Middleware
def correlation_id_middleware(request, call_next):
trace_id = request.headers.get("X-Trace-Id", str(uuid4()))
with logger.contextualize(trace_id=trace_id):
return call_next(request)
Now every log inside that request automatically includes trace_id. No threading through every function.
Step 4: Metrics (15 min)
Counters, gauges, histograms:
from prometheus_client import Counter, Histogram
requests_total = Counter("requests_total", "Total requests", ["method", "endpoint"])
request_duration = Histogram("request_duration_seconds", "Request duration")
@app.middleware("http")
async def metrics_middleware(request, call_next):
start = time.time()
response = await call_next(request)
requests_total.labels(method=request.method, endpoint=request.url.path).inc()
request_duration.observe(time.time() - start)
return response
Dashboards. Alerts. Trends.
Step 5: Traces (10 min)
Per-span timing across functions:
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def process_order(order_id):
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
validate(order_id)
charge(order_id)
ship(order_id)
Visible in your tracing tool. Tells you where time goes.
Step 6: Feature Flags (15 min)
Decouple deploy from release:
if feature_flags.is_enabled("new_checkout_flow", user=user):
return new_checkout()
else:
return old_checkout()
Roll out gradually:
1% of users
Specific cohort
Internal team only
Everyone
If it breaks, flip the flag. No deploy needed.
Tools: LaunchDarkly, Unleash, ConfigCat, or homegrown.
Step 7: Kill Switches (10 min)
For dangerous features, a master off:
if feature_flags.is_enabled("disable_new_charge_flow"):
return legacy_charge_flow()
When the new flow misbehaves: flip; investigate. No deploy required.
Critical for high-stakes systems.
Step 8: Health and Readiness Endpoints (10 min)
@app.get("/healthz")
def liveness():
return {"status": "ok"}
@app.get("/readyz")
def readiness():
return {"status": "ok", "db": db.is_connected(), "cache": cache.is_connected()}
Liveness: am I alive? (restart if not)
Readiness: am I ready for traffic? (LB skips if not)
Critical for Kubernetes, autoscaling, deploys.
Step 9: Debug Endpoints (10 min)
For ad-hoc inspection:
@app.get("/debug/config")
def show_config():
return get_safe_config() # don't expose secrets!
@app.get("/debug/metrics")
def show_metrics():
return get_metrics_snapshot()
@app.post("/debug/log-level")
def set_log_level(level: str):
logging.getLogger().setLevel(level)
Auth-protected. Only for engineers. But: invaluable at 3am.
Step 10: Document the Tools (10 min)
A runbook:
# Debugging My Service
## Logs
Search Datadog: service:my-service trace_id:<id>
## Traces
Jaeger UI: https://...
## Feature flags
LaunchDarkly: https://...
## Common operations
- Disable a feature: flip flag X
- Drain a node: hit /debug/drain
- Force GC: hit /debug/gc
## Common errors
- "DB connection failed" → check security group rules
- "OOM" → restart, then profile
Without docs, debugging tools sit unused.
What You Just Did
Built-in debugging: structured logs, levels, context, metrics, traces, feature flags, kill switches, health checks, debug endpoints, runbooks. Code designed for the bad day.
Common Failure Modes
No structured logs. Production debugging by grep.
No metrics. "Is it slow?" → no idea.
No feature flags. Every change is a deploy + risk.
Health check that's too shallow. Reports healthy when broken.
Debug endpoints unauthenticated. Attackers love them.
Continue the Debugging Systematically path
Previous — Part 9: Write a Bug Postmortem
Part of the Debugging Systematically learning path.
