Risk Assessment for Software Changes
The change request said Risk: Low. It was a one-line fix: a statement-generation service rounded interest to two decimal places in the wrong order, and a customer had noticed a one-cent discrepancy. The engineer asked an AI assistant for the fix, the tests passed, the reviewer approved it in four minutes, and it shipped on a Tuesday.
Nothing broke on Tuesday. On the first business day of the next month, the general-ledger export (a nightly job owned by finance, reading the same field as text) produced totals that no longer tied out against the bank's core system. Finance spent nine days reconciling by hand before anyone connected the discrepancy to a change that had shipped three weeks earlier and was, by every measure the change process used, low risk.
This example is a composite, drawn from patterns common to financial services change processes. The details are illustrative.
Nobody in that story was careless. The process did exactly what it was built to do: it asked the author to rate the change, and the author rated it honestly. The failure is that the Low/Medium/High field measures the author's confidence in the code, and the code was fine. What failed was a consumer nobody had asked about.
Why the label feels like enough
A risk field is attractive because it is cheap and it looks like control. Every change gets a rating, the rating is auditable, and reviewers can sort by it. In regulated organisations it also satisfies the letter of most change-management policies.
The research on heavier change control is not encouraging for the label. The 2019 Accelerate State of DevOps report found that requiring external approval (a change advisory board or a senior manager) was negatively correlated with lead time, deployment frequency and time to restore, and had no correlation with change failure rate. Approval did not make changes safer; it made them slower and larger. A rating that feeds that approval inherits the same problem.
Meanwhile the stakes for getting change risk right are high. Google's SRE book reports that roughly 70% of outages are due to changes in a live system. If most of your incidents start as changes, the point in the process where someone thinks about a change is your best leverage, and a dropdown is a poor use of it.
What the label misses: the author can only rate what the author can see
Walk the rounding fix through the chain that produced the incident:
Assumption (the field is only read by the statement renderer) → Decision (rate it Low; review the diff) → Hidden dependency (the GL export parses the same field as text, and the new rounding changes its string form for about one value in two hundred) → Failure mode (totals drift by cents across hundreds of thousands of accounts) → Business consequence (nine days of manual reconciliation in a close period, and an audit finding).
Every link in that chain was invisible from the diff. The author knew the code. The reviewer knew the code. Neither knew who else read the data, and the risk field never asked.
This is a discovery failure, not a coding failure. The one question that would have caught it is the question a business analyst asks at the start of any requirements conversation: who else depends on this? Risk assessment is that question asked again at the moment of change.
AI assistance makes the gap wider, not narrower. When an assistant writes the change, the author's confidence comes partly from the tool, and the author's knowledge of the surrounding system is no deeper than before. The 2024 DORA report found that a 25% increase in AI adoption was associated with an estimated 7.2% decrease in delivery stability. The report's authors connect this to fundamentals such as batch size and testing rather than to the tools themselves, which is the point: faster authoring puts more weight on the step that checks what the author couldn't see.
How a useful assessment works: four questions, one of them about data
A change risk assessment answers four questions. The output is a short paragraph or table that a reviewer can disagree with, not a label.
Blast radius: who or what is affected if this does the worst plausible thing for thirty minutes? Name populations, not services: "all authenticated web users", "finance's month-end GL export", "partners on API v2".
Failure modes: what specifically could go wrong? The categories that recur are the change being wrong; the change meeting real data it was never tested against; an unknown consumer breaking; a latent bug exposed by a newly exercised path; the rollout failing; and the rollback failing.
Likelihood, in three buckets: has happened to changes like this, plausible given how the system is built, would need a coincidence. Precision beyond that is false.
Impact on four axes: customer-visible effect, time to detect, recoverability (lost data and financial errors recover far worse than an outage) and spread to other systems or partners.
The rounding fix fails at question one. "Statement renderer only" is the author's model of the blast radius, and nobody tested it. That is why the most important line in a good assessment is usually the consumers line, and why it should be filled in from evidence rather than memory.
Walking the fix through the assessment
Here is the same change assessed properly. It took the author about twenty minutes, most of it spent on one search.
The discovery move. Before rating anything, the author searches for every reader of the field: a repository-wide search for the column name, plus a query against the warehouse's lineage or the database's access logs for jobs that select it. The repository search finds the renderer. The warehouse query finds a second reader, gl_export_nightly, owned by finance.
## Risk assessment: CR-4471, interest rounding order
Consumers of statement_line.interest_amount (found by repo search + warehouse lineage):
- statement-renderer (ours)
- gl_export_nightly (finance) reads the value as TEXT, sums in the export
Blast radius: all retail statements (~410k accounts/month); finance month-end GL tie-out.
Failure modes:
- New rounding changes string form for some values -> GL totals drift.
Likelihood: plausible (export parses text). Impact: financial error, slow to detect
(surfaces at month-end), recoverable only by manual reconciliation.
- Rounding fix itself wrong for negative interest (adjustments).
Likelihood: plausible. Impact: customer-visible, recoverable.
Mitigation: run gl_export against a copy of last month's data with and without the change;
diff totals. Add a negative-interest case. Notify finance owner before release.
Net: Medium, driven by the export. Ship after the export diff is clean.
The export diff takes an hour and shows 2,140 lines that change. Finance adjusts the export to read the numeric column, the change ships a week later, and month-end ties out. The code in the final release is identical to the code that would have shipped on Tuesday. The assessment changed the plan, and that is the only reason to do one.
How to tell whether your current process measures anything
You can test your own risk field this week with data you already have.
The label calibration test. Take your last 20 incidents that were caused by a change. For each, look up the risk rating on the change that caused it. Count how many were rated Low.
If most change-caused incidents came from Low-rated changes, your label is measuring author confidence, not risk. That is the most common result.
If the incidents are spread evenly across Low, Medium and High, the label carries no information at all and the review time attached to it is overhead.
If incidents cluster in High-rated changes that shipped anyway, the label works but the response doesn't: you are rating risk and then not acting on it.
Two more signals worth counting: the share of change requests whose risk text is identical to another request (boilerplate), and the share of High-rated changes that shipped with no mitigation listed. Either number above about a third is a process producing paperwork rather than judgement; treat that threshold as a rule of thumb, not a benchmark.
Responding to what the assessment finds
Risk is an input to a decision about how to ship, not whether to approve.
Likelihood × impact | Response |
Unlikely, low impact | Ship normally. More process costs more than the risk. |
Unlikely, high impact | Reduce exposure: feature flag, canary, a named person watching the dashboards for the first hour. |
Plausible, low impact | Ship if the value justifies it, and plan for the small failures. |
Plausible, high impact | Don't ship as-is. Reduce likelihood (a test against real data, a design change) or reduce impact (smaller scope, slower rollout, dual-write). |
Two patterns are systematically under-rated and deserve a bump of one level by default: anything that changes stored data or its format, and anything the team has never done before. Two are systematically over-rated: additive, backward-compatible schema changes, and changes that pattern-match to scary categories but follow a rehearsed path.
Where the assessment changes, and where it isn't needed
Standard changes don't need one each time. ITIL 4's change enablement practice distinguishes standard changes (pre-authorised, low-risk, repeatable) from normal and emergency changes. A deploy that follows a rehearsed pipeline to a stable service was assessed once, as a pattern. Re-assessing each instance trains people to write boilerplate. The line is the pattern boundary: the moment a change touches something the pattern didn't cover, such as a migration or a new consumer, it is a normal change again.
Emergency changes get the assessment afterwards. During an incident, the assessment is the post-incident review. Write it within a day, using the same four questions, so the calibration data stays complete.
Combined releases need one assessment for the bundle. Five individually Low changes shipped together can interact. If your release train batches changes, assess the train.
Authors are not the only assessors. For data-format changes, the owners of downstream consumers should see the assessment before release. That is a notification, not an approval gate, and it is the cheapest control in this article.
Installing it without adding a meeting
Replace the dropdown with a template containing four headings and a mandatory Consumers line, with a note on how they were found (search, lineage, logs).
Make the reviewer responsible for disagreeing. A reviewer who approves a change is approving the assessment too; a reviewer who can't find the blast radius in the text sends it back.
Join incidents to change requests. Every post-incident review records the change that caused it and whether its assessment named the failure mode. That join is your calibration data.
Run the label calibration test quarterly and publish the result to the teams. Calibration improves when people see their predictions next to outcomes.
Retire the assessment for patterns that prove safe. If a change type produces no incidents across a quarter of volume, promote it to a standard change.
How the practice itself fails
The defensive assessment. If High ratings attract delay and Low ratings attract nothing, authors learn to write Low. Keep High ratings cheap by tying them to mitigations rather than to extra approvals.
The assessment that changes nothing. If the plan after the assessment is identical to the plan before it, either the change really was routine (make it a standard change) or nobody acted on what was found.
The search that wasn't run. A Consumers line written from memory is the original dropdown with more words. Ask how it was found.
Final takeaway
A risk label records how confident the author is, and authors are confident about the code they can see. Incidents come from what they can't see: consumers, data, rollout paths. A useful assessment asks the discovery question again at the moment of change, finds the answer with evidence, and changes how the change ships.
Your next action: run the label calibration test on your last 20 change-caused incidents. If more than half were rated Low, add a mandatory Consumers line to your change template this sprint. For how changes move through approval once assessed, see running a change advisory board that doesn't slow you down; to measure whether it's working, start with instrumenting change failure rate. If you want a place to track the risks that outlive a single change, the risk register template is built for it.
Sources
Beyer, B., Jones, C., Petoff, J., Murphy, N. R. (eds.), Site Reliability Engineering: How Google Runs Production Systems, O'Reilly (2016), Chapter 1: "SRE has found that roughly 70% of outages are due to changes in a live system." https://sre.google/sre-book/introduction/
DORA / Google Cloud, Accelerate State of DevOps Report 2019: external change approval negatively correlated with lead time, deployment frequency and restore time; no correlation with change failure rate. https://dora.dev/research/2019/dora-report/
DORA / Google Cloud, Accelerate State of DevOps Report 2024 (October 2024): a 25% increase in AI adoption associated with an estimated 7.2% decrease in delivery stability and 1.5% decrease in throughput. https://dora.dev/research/2024/dora-report/
AXELOS, ITIL 4 Practice Guide: Change Enablement (2020): standard, normal and emergency change types. https://www.axelos.com/certifications/itil-service-management
Part of the Software Quality Engineering guide — ShiftQuality's complete map to building quality in.
