top of page

Quality Gates Worksheet

S. West
1 hour ago
5 min read

Quality Gates Worksheet

The scanner caught the leaked API key on the day it was committed. Three weeks later the key was still in the repository, because the scanner was only allowed to warn.

A team ran secret detection on every pull request. It flagged the key correctly, posted a yellow badge nobody opened, and the merge went ahead because the check wasn't required. A routine credential rotation found the key, and the follow-up showed four other warned-on secrets that quarter, all merged.

(Composite example, assembled from patterns common in teams whose CI checks report but don't enforce. The figures are illustrative.)

Most teams have quality gates. Many of those gates do not gate. This worksheet starts from the inventory end: list every check that runs today, find out which can actually stop a change, score the keepers, add the missing ones in order, and decide in advance what happens when one fires under pressure.

Step 1: audit your existing gates

List every gate that runs today and answer each column honestly.

Gate

Blocks or only observes?

Last time it caught a real problem

Required but bypassable?

Can someone say why it exists?

______

☐ Blocks ☐ Observes

______

☐ Yes ☐ No

☐ Yes ☐ No

______

☐ Blocks ☐ Observes

______

☐ Yes ☐ No

☐ Yes ☐ No

______

☐ Blocks ☐ Observes

______

☐ Yes ☐ No

☐ Yes ☐ No

______

☐ Blocks ☐ Observes

______

☐ Yes ☐ No

☐ Yes ☐ No

The "always green" test. For each gate marked required but bypassable, pull its history and find the most recent override. Has it held anything back since then? A check that runs on every change and has never stopped one is a status light, not a gate.

The remove-it test. For any gate nobody can explain, ask what would happen if you deleted it. If no one has an answer, and no regulation or contract requires it, remove it. Checks that nobody trusts make people ignore the ones that matter.

Step 2: score each gate you keep

Score each quality 0 (fails), 1 (partial) or 2 (solid).

Quality

What solid looks like

Score

Specificity

It catches one named class of problem, such as untested new code in the diff, rather than a vague quality score

☐ 0 ☐ 1 ☐ 2

Calibration

It doesn't fire on ordinary work and it isn't silent forever. Tuned with the team (in our experience, expect a few months of iteration)

☐ 0 ☐ 1 ☐ 2

Speed

It adds a few minutes, not most of an hour. Slow checks get skipped

☐ 0 ☐ 1 ☐ 2

Explainability

A failing engineer knows why within moments. For example, "complexity 18 exceeds threshold of 15; extract the helper in lines 42–67" beats "policy violation"

☐ 0 ☐ 1 ☐ 2

Visible intent

The team knows what the gate protects and would defend it

☐ 0 ☐ 1 ☐ 2

Total: ______ out of 10. 8–10: worth defending. 4–7: fix the low scores before you rely on it. 0–3: it will drift into theatre, so recalibrate or retire it.

Step 3: build the missing gates in this order

Tick each one when it is enforced by the platform, not by a policy someone can waive.

  • ☐ 1. Block on test failure: required CI before merge and before deploy. If a red build can still reach production, the tests aren't gating anything.

  • ☐ 2. Branch protection: no direct commits to main; changes arrive by pull request with required reviews and an audit trail. (Minutes to configure on GitHub/GitLab. Turn off the default admin bypass, or the gate is bypassable by design.)

  • ☐ 3. Security scanning that blocks: static analysis, dependency scanning and secret detection. The bar is no new High or Critical finding (the CVSS ratings) in the changed code, not a spotless codebase.

  • ☐ 4. Schema and migration review: database changes go to a reviewer who knows the data layer.

  • ☐ 5. Observability requirements: every new endpoint ships with at least one metric, one log and one trace.

  • ☐ 6. Performance budgets: the change shows it stays within an agreed budget on representative benchmarks.

In our experience, a team that genuinely enforces the first four is already ahead of most teams.

Step 4: decide the bypass rule before you need it

A well-built gate will eventually fire on a change that someone senior wants out today. Agree now which path your team takes.

  • ☐ Fix and ship through: fix the problem, then ship the change through the gate. This keeps the gate credible.

  • ☐ Bypass and queue the fix: override, ship, and promise a fix later. This tells everyone the gate is optional, and that lesson can stick for years.

When a gate fires under pressure

Your answer

Who may override, if anyone?

______

What must be recorded when they do?

______

Who reviews overrides, and how often?

______

Which gates may never be overridden?

______

The person with authority to override, visibly choosing not to, does more for gate discipline than any configuration change. The cost of weakening a gate is real but arrives later, as an incident nobody can trace back; the cost of missing a ship date is visible but usually smaller over the long arc.

A gate is only as strong as the leader's willingness to be held by it. In our experience the difference shows up later, in the incident rate and in whether the team still trusts its gates.

How to use it

  • Who: the engineering lead who owns the pipeline, with one engineer who knows the CI configuration and whoever holds admin rights on the repository. An hour for a first pass on one repository.

  • Output: a gate inventory with block or observe marked, a score for every gate you keep, a dated plan for the missing gates, and a written bypass rule.

  • Done when: every gate either blocks at the platform layer or has been retired, no keeper scores below 4, and the bypass rule names who may override and where the record goes.

Where this changes

A small team with one repository and a handful of contributors can run Steps 1 and 3 in a single sitting and keep the bypass rule to one line. A regulated or large organisation needs more: gates mapped to the controls they evidence, override records kept for audit, and separation between the people who configure gates and the people whose changes they check.

The reasoning behind this worksheet, including how gates erode one waiver at a time, is in quality gates that actually gate. For the fastest gate in the pipeline, read smoke testing as a five-minute quality gate. For AI features, see quality gates for shipping AI. To track override and block rates over time, pair this with quality metrics that actually matter.

Final takeaway

A check that can't stop a change is a report, however green it looks. Find out which of your gates block, score and fix the keepers, and settle the bypass rule while nobody is under deadline.

Sources

bottom of page