Integration Testing: When, How, and Why

Shawn West
Mar 7
9 min read

Updated: Aug 10

Two services, each with a green unit suite, that broke the instant their responses had to cross the wire between them. The bug wasn't in either service — it lived in the gap. Here's the mechanism that puts integration bugs at boundaries, one seam walked to the test that catches it, and a way to find the untested seams already sitting in your own system.

The order service was green. Ninety-odd unit tests, all passing, coverage report the shade of green that ends a code review. The inventory service it called was green too — its own suite exercised the reservation logic, the shortfall reporting, the edge cases. Both teams shipped. The seam between them had never been tested by anything, and that is where it broke.

Here is how it surfaced. Customers with perfectly in-stock carts started getting checkout failures — 500s on the happy path, the one flow nobody worried about. Carts that were partly out of stock, the messy case both teams had carefully tested, went through fine. It was backwards from every intuition. The failing requests had one thing in common: every item was available. And the two services responsible had, between them, exactly zero tests where a real response actually travelled from one to the other.

This is the class of bug unit tests structurally cannot see. Not because the unit tests were bad — they were good, and good unit tests are worth writing — but because a unit test isolates a component by definition, which means it replaces everything on the other side of the boundary with something the test author wrote. The mock returns what the author expected. Production returns what the other team actually built. When those two drift apart, no amount of unit coverage on either side will tell you, because neither side is ever looking at the other.

Why the bugs live at the boundary

Integration bugs cluster at boundaries for a reason that is almost geometric: a boundary is the one place in your system where two sets of assumptions have to line up, and it is the one place no single component can verify alone. Inside a component, the compiler and your unit tests keep the assumptions honest. The moment a value crosses a process, a network hop, or a serialization step, it leaves the reach of both.

Walk the specific categories, because "components don't integrate" is a slogan until you can name where it bites:

Contract drift. The caller expects a field; the callee renamed it, made it optional, or nested it one level deeper. Each side's tests encode its own view of the contract, so both stay green while the two views diverge.
Serialization. An object that round-trips perfectly inside one service loses something crossing the wire — a datetime becomes a string, a Decimal becomes a float, an empty collection gets omitted entirely by a serializer tuned to keep payloads small. The in-memory object and the wire representation are not the same object.
Null versus empty across a call. Inside one language, null, [], and "field absent" are three distinct things you handle deliberately. Across a JSON boundary they blur, and the two sides often disagree about which one means "nothing happened."
Transaction boundaries. A unit test with a mocked repository never commits. Two real operations against a real database can interleave, half-commit, or hold a lock the mock never modelled.
Ordering and timing. A publishes before B is ready; a message arrives twice; an async result lands after the assertion already ran. Mocks resolve instantly and in order, so the mock world is one where these problems do not exist.

Every one of these lives between components, in the space unit tests are designed to abstract away. That's the whole point of the isolation — and the whole reason a green unit suite on each side proves nothing about the join.

One seam, walked to the bug

(Developed example — composite scenario.)

Take the order/inventory seam concretely. An OrderService owns checkout; an InventoryService owns stock. At checkout, the order service asks inventory to reserve the cart's items and gets back a reservation plus a list of shortfalls — items it couldn't fully reserve. If anything is short, the order is rejected.

The inventory service models its response and, to keep payloads lean, serializes with empties dropped:

# inventory_service.py  (the callee)
class ReserveResult(BaseModel):
    reservation_id: str
    shortfalls: list[Shortfall] = []   # empty when everything reserved cleanly

@app.post("/reservations")
def reserve(req: ReserveRequest) -> dict:
    result = ReserveResult(
        reservation_id=new_id(),
        shortfalls=compute_shortfalls(req),
    )
    # tuned for small payloads: default-valued fields are omitted
    return result.model_dump(exclude_defaults=True)

Its unit test is green, and reasonably so:

def test_reserve_reports_shortfalls():
    # 100 requested, 3 in stock -> a shortfall
    result = reserve(ReserveRequest(items=[Item(sku="A", qty=100)]))
    assert result["shortfalls"][0]["sku"] == "A"

That test always exercises the path where shortfalls is non-empty, so the field is always present. The bug is invisible from here.

The order service consumes that response:

# order_service.py  (the caller)
def checkout(cart, inventory_client) -> Order:
    payload = inventory_client.reserve(cart.items)
    if payload["shortfalls"]:              # any item short -> reject
        raise OutOfStock(payload["shortfalls"])
    return persist_order(cart, payload["reservation_id"])

Its unit test is also green, and also reasonable:

def test_checkout_places_order_when_stock_available():
    stub = StubInventory(returns={"reservation_id": "r_1", "shortfalls": []})
    order = checkout(cart_of("A", qty=2), stub)
    assert order.status == "placed"

Look at what the stub does: it hand-writes "shortfalls": [] into the response. The author knew the field should be there, so they put it there. The stub encodes the caller's belief about the contract, not the callee's actual output.

Now run the real thing. A fully in-stock cart reaches inventory. compute_shortfalls returns []. exclude_defaults=True sees a default value and drops the key. The JSON on the wire is:

{ "reservation_id": "r_1" }

The order service does payload["shortfalls"] on a dict that has no shortfalls key. KeyError. Five hundred. And note which orders die: only the ones where nothing was short — the happy path — because that's the only path that produces an empty list to be dropped. The unhappy path both teams tested carefully sails through, because a shortfall keeps the field present. The suite is green on exactly the cases that work in production and silent on the case that doesn't.

Here is the test that would have caught it — an integration test that runs both real sides with a real response crossing between them:

def test_checkout_reserves_real_inventory_for_in_stock_cart():
    inventory = start_inventory(stock={"A": 10})     # the real app
    client = InventoryClient(base_url=inventory.url)  # real HTTP, real serialization
    cart = cart_of("A", qty=2)

    order = checkout(cart, client)

    assert order.status == "placed"
    assert inventory.reserved("A") == 2

Nothing is stubbed at the seam. The real serializer runs, the empty array actually gets dropped, the real client parses the actual bytes, and checkout throws the KeyError — in CI, on a plain in-stock cart, before a customer ever sees it. The test doesn't just fail; it fails pointing at the boundary, which is the whole value. One test crossing the seam surfaces a bug that no quantity of tests on either side of it could.

The discovery move that would have prevented it is worth naming, because it's cheaper than any test: the two teams never wrote down the response contract — specifically, whether an empty shortfalls is sent as [] or omitted. Each team answered that question privately, in code, and answered it differently. The seam bug is a discovery gap wearing a serialization costume. If the contract had been agreed and shared, the serializer config would have been a caught review comment instead of a production incident. Contract testing exists precisely to make that shared agreement executable.

Find your own untested seams

The order/inventory story is only useful if you can run its diagnostic on your own system this week. You can, and the signal is countable.

List every seam. A seam is any place a value in your service crosses into something you don't control in-process: each outbound HTTP client, each database write path, each queue publish and each consumer, each call into another team's library that talks to a network. Grep for them — the client classes, the repository/adapter layer, the publishers. You'll get a finite list; most services have between five and fifteen.

For each seam, ask one observable question: is there a test where both sides are real and data actually crosses — not a test where this side runs and the other side is a mock? Cross-reference your seam list against the tests that instantiate the real dependency (a real test database, a real in-process app under TestClient, a real broker with a test topic). A seam with a real crossing test is covered. A seam where every test replaces the other side with a double is an untested seam, no matter how green the unit suite around it looks.

Count them. The number you want is seams with zero crossing tests. That integer is your real integration exposure, and it is usually higher than the team believes, because coverage tools count lines executed, not boundaries crossed — you can have 90% line coverage and 0% seam coverage. Rank the untested seams by blast radius (a payment or inventory boundary outranks a logging one) and write the first crossing test for the seam at the top. You are not chasing a coverage number; you are converting invisible boundaries into ones something actually looks at.

What to make real, and what to fake

"Cross the seam with real components" does not mean stand up your entire production topology in CI. The skill is choosing, per dependency, what is real and what is a double — and being honest that a double at a boundary reintroduces exactly the drift risk you just saw. Every double is a place your test encodes an assumption the real system might not honour. That's the tradeoff, stated plainly:

Dependency at the seam	Make it…	Why	Cost you accept
Your own database	Real (test instance)	The schema, types, and constraints are the behaviour; a fake DB tests fiction	Slower tests; isolation work
A service your team owns	Real, in-process	Cheap to run, and it's your contract to keep honest	Some setup wiring
A message broker	Real with test topics	Ordering and delivery semantics are the thing under test	Broker in the test env
Expensive infra (object store, email)	Fake (in-memory equivalent)	Behaviour is simple and well-understood; realism isn't worth the cost	Fake can drift from real
A third party's own internals	Stub + a contract test	You can't and shouldn't test their system; you test the agreed shape	Stub can lie if contract drifts

The rule underneath the table: use a real dependency wherever the interaction itself is what could break, and a double only where the interaction is well-understood and the realism isn't worth the cost. The dangerous move is stubbing the seam that carries the risk — which is what the order service did. Choosing among mock, stub, and fake is a decision with real consequences at each boundary; the differences between mocking, stubbing, and faking decide what your test actually proves versus merely asserts.

When an integration test is the wrong tool

Reaching for an integration test reflexively is its own failure. Three cases where it's the wrong instrument:

Pure logic with many branches. A pricing calculation with twenty cases doesn't need a database round-trip per case; it needs twenty fast unit tests. Paying network and setup cost to exercise branching logic is slow for no added truth — the logic has no seam.
A third party's internal behaviour. You are not testing whether Stripe charges a card correctly; that's their suite. You test that you produce and consume the agreed format. That's a contract test plus a stub, not a live call — calling the real third party makes your CI depend on their uptime and rate limits to tell you whether your code is right.
Async and stateful edges, tested naively. A seam that involves eventual consistency, retries, or out-of-order delivery will produce a flaky integration test if you write it with a naive fixed sleep. That's not a reason to skip the test; it's a reason to test the hard parts — async, external dependencies, and state with proper waits and controlled clocks rather than pretending timing doesn't exist. A flaky seam test that gets ignored is worse than none, because it trains the team to disbelieve red.

The default is still: if two components have to agree and the agreement could break, test the agreement with both sides real. The exceptions are where there is no genuine seam (pure logic), where the seam isn't yours to test (third-party internals), or where the seam needs more care than a straight-line test can give (async state).

Run the audit

Don't start by writing tests. Start by counting seams. This week, take one service, list every place it crosses a boundary you don't control in-process, and for each one answer the single question: is there a test where both sides are real and data actually crosses? The seams that answer "no" are your integration exposure — the joins currently held together by two green suites that have never once looked at each other.

Pick the highest-stakes "no" — the payment call, the inventory reservation, the write path that money or correctness depends on — and write one crossing test for it, with the real dependency on the other side and nothing stubbed at the seam itself. If it goes green on the first run, you've documented a contract. If it goes red, you've just found in CI the bug that was going to find you in production, on the happy path, on the flow nobody was worried about.

ShiftQuality