Profile a Slow Endpoint

Performance Engineering · Part 1

The first rule of performance work is the one everyone breaks: don't guess. Almost every developer's intuition about why code is slow is wrong, so hours get burned optimizing the fast part while the real culprit sits untouched. A profiler tells you the truth. This walks through profiling a slow endpoint to find where the time actually goes — spans, flame graphs, and the usual suspects: a database call, an N+1, a blocking HTTP request.

"My endpoint is slow." First step: find out where the time goes. Guessing wastes hours.

Step 1: Measure (5 min)

Add timing:

import time

@app.route('/slow-endpoint')
def slow_endpoint():
    start = time.time()
    
    # Your endpoint logic
    result = do_work()
    
    elapsed = time.time() - start
    print(f"Total: {elapsed:.3f}s")
    return result

Establish baseline. Is it 500ms or 5s?

Step 2: Add Spans (10 min)

Wrap sections:

@app.route('/slow-endpoint')
def slow_endpoint():
    timings = {}
    
    t = time.time()
    user = db.get_user(request.user_id)
    timings["get_user"] = time.time() - t
    
    t = time.time()
    items = db.get_items(user.id)
    timings["get_items"] = time.time() - t
    
    t = time.time()
    enriched = enrich(items)
    timings["enrich"] = time.time() - t
    
    print(timings)
    return jsonify(enriched)

Reveals which section is slow. 4 seconds in enrich(); now you know where to look.

Step 3: Use a Profiler (15 min)

Manual timing has limits. Use a profiler.

Python (cProfile):

import cProfile

cProfile.run('slow_endpoint()', sort='cumulative')

Outputs every function with time spent.

py-spy for production:

py-spy record -o profile.svg --pid PID

Flame graph; no code changes; works in production.

Step 4: Read a Flame Graph (10 min)

Flame graph:

• X-axis: time

• Y-axis: stack depth

• Width: time spent

Wide boxes = bottlenecks. Find them at the top of stacks.

Common patterns:

• Wide DB call boxes → slow query

• Wide HTTP client boxes → external service slow

• Many tiny boxes → CPU-bound work or N+1

Step 5: Database Profiling (10 min)

For DB-heavy endpoints:

EXPLAIN ANALYZE SELECT ...;

Shows the plan and timing. Look for:

• Sequential scans on big tables

• Many rows returned then filtered

• Nested loops with high iteration

In your code:

db.execute("EXPLAIN ANALYZE " + query, params)

Capture in development; tune the query.

Step 6: HTTP Client Profiling (5 min)

External API calls:

import requests
from time import time

t = time()
response = requests.get('https://external.com/api/data')
elapsed = time() - t
log.info(f"External call took {elapsed:.3f}s")

If external is 500ms and you're calling 10 times, that's 5s right there. Parallelize or cache.

Step 7: Memory Profiling (10 min)

Slow can be memory pressure (paging, GC). Check:

import tracemalloc

tracemalloc.start()
result = slow_function()
snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics('lineno')[:10]:
    print(stat)

Shows where memory is allocated.

Memory leaks accumulate → slow GC pauses → endpoint slowness.

Step 8: Distributed Tracing (covered in observability) (5 min)

For microservices:

with tracer.start_as_current_span("db_query"):
    user = db.get_user(...)

with tracer.start_as_current_span("call_other_service"):
    data = external.get(...)

Each span recorded; visible in tracing UI.

Most useful when the slow path crosses service boundaries.

Step 9: Common Findings (10 min)

Patterns you'll see:

1. N+1 queries: 1 query + N more (tutorial 2)

2. Missing index: seq scan on big table (tutorial 3)

3. External API blocking: sync call to slow vendor

4. Cache miss: every request hits DB (tutorial 4)

5. GC pause: Java/Python pause for memory cleanup

6. Lock contention: many threads waiting on a lock

Most issues fall into these categories.

Step 10: Iterate (10 min)

Profile → fix biggest → measure → repeat.

Don't fix things that don't matter. The first optimization usually delivers 80% of the gain; subsequent ones get marginal.

Stop when you hit the latency target. Premature optimization is real.

What You Just Did

You can find where time goes in a slow endpoint. Profilers; flame graphs; database EXPLAIN. The foundation for everything else.

Common Failure Modes

Guessing instead of measuring. "I think it's the DB." Wrong half the time.

Optimizing what's already fast. 0.1% time on inner loop; 99% on one DB query.

Profiler overhead. Sampling profilers are cheap; tracing profilers can be heavy.

Profile in dev only. Production traffic differs; profile production (carefully).

Fixing one bottleneck; another appears. Iterate.

---

Continue the Performance Engineering path

• Next — Part 2: Find N+1 Queries

Part of the Performance Engineering learning path.