Change Impact Analysis: A Step-by-Step Guide

Impact analysis is usually done by looking at the change and asking what it affects. That direction finds the easy half. The expensive half is only visible if you run the query the other way.

Aldermoor Health changed a token-expiry setting on its auth service — three lines of configuration, reviewed, approved, with an impact analysis attached.

The analysis listed three services. All three were correct. All three were fine.

The mobile app was down for four hours.

The mobile app didn't appear because the person doing the analysis was working outward from the change: what does this touch? The auth service doesn't call the mobile app. The mobile app calls it — and holds tokens for longer than the new expiry allowed. From the change's point of view, the mobile app was invisible.

Two directions, and only one of them is memory-safe

This is the whole mechanism, and it's why competent people miss things.

• Forward — what does this change touch? What does it call, write to, publish? Answerable from the change itself, by the person making it. Reliable.

• Backward — what depends on the thing I'm changing? Who calls it, reads its data, parses its output, has a hard-coded assumption about it? Not answerable from the change, and usually not answerable from memory either.

Forward analysis is a reading exercise. Backward analysis is a query, and if you don't run one you are substituting recall — and recall is systematically biased toward the consumers you personally work with.

Aldermoor's analyst knew of three consumers. There were nine. Six belonged to teams they'd never worked with, which is precisely why they didn't come to mind.

The test you can run: take your last impact analysis and ask which direction produced each entry. If everything on the list is something the change touches, you have done half of it.

Running the backward query

The specific instrument doesn't matter as much as running one rather than asking people. In rough order of reliability:

Source	Finds	Misses
Access logs / API gateway	Every caller in the log window, including ones nobody remembers	Consumers that are dormant but alive — the monthly batch job
Service mesh / trace data	Live call graph, with volumes	Anything not routed through the mesh
Code search across repos	Hard-coded references, config files	Other people's repos, third parties
Database grants	Who reads the table directly	Nothing that goes through your API
Asking around	Context the tools miss	Everyone you didn't think to ask

The last row is the one to be careful with. Asking is genuinely useful for the things logs can't see — a team mid-migration, a customer with an unusual integration — but it must come after the query, as a supplement. Done first, it anchors the analysis on the consumers you already know about.

One caution on log windows: a 7-day window misses monthly jobs. Aldermoor now uses 90 days for anything touching an interface, which is how they later found a quarterly reconciliation script nobody had thought about in two years.

The test: for the service you're changing, run the caller query over 90 days and compare it against the list you'd have written from memory. The difference is your blind spot, and it's usually not small.

The dependencies that aren't code

Once the backward query is running, the remaining misses cluster in three places that no call graph shows:

• Data consumers. Reports, dashboards, an analyst's spreadsheet pulling from a replica. Changing a column name breaks these silently — no error, just a quietly empty chart. These are found by asking whoever owns reporting, and nowhere else.

• Humans with an assumption. Support has a script that says "tell the customer to wait five minutes." A runbook references a screen you're removing. These break the moment you ship and surface as confusion rather than an alert.

• Contracts and timing. Anything with an expiry, a schedule, or a retry interval — the failure mode from the opening story. The mobile app wasn't wrong; it was built against a promise that changed.

The pattern across all three: the code call graph shows what talks to you, not what believes something about you. Beliefs are held by people, documents and cached values, and they don't appear in traces.

The test: for your change, ask what someone might believe about this system that will stop being true. If the answer is anything, that belief has a holder who needs telling.

Assess per touchpoint, not per change

Once you have the list, the instinct is to give the change a single rating — high, medium, low — and move on. That number gets negotiated and tells you nothing.

Per touchpoint, three questions produce actions instead:

1. Does it break, degrade, or is it fine? Be specific. "Might be affected" is not an assessment.

2. Would we know? A break with an alert is very different from a break the analyst discovers next Thursday. Silent failures deserve to be treated as more severe than loud ones, which is the reverse of how they usually get ranked.

3. What's the action? Notify, sequence before, sequence after, or nothing. Every touchpoint gets one of the four.

The mobile app in the opening story is a "breaks, silently, needs sequencing before" — the client had to ship a token-refresh fix first. That's not a warning to be circulated; it's a dependency that reorders the work.

The test: for each touchpoint, name the action. Any entry sitting at "monitor" is usually one nobody knew what to do with.

When not to bother

Impact analysis is proportional to blast radius, and running it on everything trains people to run it badly.

• The change is behind a flag, off by default. The impact analysis belongs at the flag flip, not the merge.

• It's contained within one service with no interface change. The call graph is your team's, and code review covers it.

• It's a standard change. The analysis was done once at promotion; that's what standard means.

• It's a genuine emergency. You're accepting the risk deliberately — but then it belongs in the review afterwards, because that's where the missed dependency turns up.

Where it always earns its cost: interface changes, shared-data changes, anything with a timing or expiry semantic, and anything decommissioned. It is one of the four things a change process actually needs.

The test: if the change alters something another team could hold a belief about, run it. Otherwise don't.

What to change this week

Pick the service you're most likely to change next. Run the caller query over 90 days, before you need it.

Compare that list to the one you'd have written from memory. Then, for the consumers you didn't know about, find out who owns them — because that list is also your notification list, and it's the artifact most teams don't have when they need it at four o'clock on a Friday.

Aldermoor keeps a dependents list per service now, regenerated weekly. The auth service has eleven entries. Nine of them were a surprise the first time.