top of page

Game Day Starter Kit

S. West
1 hour ago
4 min read

The runbook said "fail over to the replica." Nobody had ever done it, and at 3am it turned out the replica was weeks behind.

The team had designed for failure on paper. There was a replica, a runbook and an alert. When the primary database went down overnight, the alert fired to a channel nobody watched after hours, the on-call engineer found the runbook forty minutes in, and the failover step pointed at a replica that had stopped replicating after a config change. Recovery took most of the night. Every piece existed; none of it had been tested together.

(Composite example, assembled from patterns common in teams that plan for failure but never rehearse it. The figures are illustrative.)

Chaos engineering replaces "we designed for failure and hope it works" with evidence. You don't need a hyperscaler's tooling. You need one scheduled game day. This kit starts from the failure you are most afraid of and works backwards to whether your detection, people and recovery actually hold up.

Game days began as Amazon's "GameDay" programme in the early 2000s, led by Jesse Robbins; chaos engineering was later formalised in the Principles of Chaos Engineering.

Step 1: know what a game day proves

A game day is a scheduled exercise where you simulate one specific failure and watch what actually happens. One game day can test all five of these at once:

  • ☐ Monitoring detects the failure.

  • ☐ Alerts reach the right people.

  • ☐ The runbook addresses the scenario.

  • ☐ The team can execute under pressure.

  • ☐ Recovery actually works.

Controlled chaos is almost always cheaper than a real incident, and the team learns to respond under low pressure rather than high.

Step 2: follow the five-step format

Step

Do this

1. Pick a scenario

One concrete failure: "the primary database becomes unavailable."

2. Schedule it

During business hours, when the team is available, not off-hours.

3. Simulate

Inject the fault, with tooling or by deliberately taking the thing down.

4. Observe and respond

The team follows the runbook. Don't rescue them: watch what breaks.

5. Debrief

What worked, what didn't and what the fix is. Write it down.

Step 3: state your hypothesis first

A game day without a hypothesis is just breaking things. Write down what you expect before you inject anything.

  • Bad: "Let's break something and see what happens."

  • Good (example): "We believe the system can lose 30% of its workers and still serve requests with an error rate of 5% or less. Let's verify." The numbers here are illustrative; set your own from your service targets.

Our hypothesis: We believe ______ can happen and the system will ______ (measured by ______), with failure defined as ______.

Then define your steady state: your normal, and the bounds the hypothesis must hold within.

Steady-state signal

Normal value

Request rate

______

Error rate

______

Response time (p95 and p99)

______

Business metric (orders, sign-ups)

______

Step 4: put the safety rails in place

Every experiment, no exceptions:

  • ☐ Blast radius bounded: start in staging, then one node, one region or off-peak, and expand (towards production) only as confidence builds.

  • ☐ Kill switch ready: you can abort and undo immediately.

  • ☐ Humans actively watching.

  • ☐ Stakeholders aware that it is happening.

Staging is a safe on-ramp, not the end state. The Principles of Chaos Engineering favour experiments in production, because that is where real traffic and real dependencies live. Get there in small, reversible steps.

Step 5: choose a failure scenario

Category

Examples

Resource

Server termination, process crash, disk full, memory exhaustion

Network

Latency injection, packet loss, dropped connections, DNS failure

Dependency

Database unavailable, slow external API, cache loss, queue lag

Software

Bad deploy, config error, bad data

Tools (optional): Chaos Toolkit, LitmusChaos (Kubernetes), Gremlin, AWS Fault Injection Service, Chaos Monkey (Netflix OSS; needs Spinnaker). For your first game day you may need none of them: just turn something off.

Step 6: when the experiment fails

If the system doesn't survive, that's the point of the exercise. Work through it in order:

  1. Stop if it is actively causing harm.

  2. Analyse why it failed.

  3. Fix the underlying weakness.

  4. Re-test with the same fault.

  5. Document the finding and the fix.

Your growing list of findings and fixes is your team's resilience history.

Step 7: debrief

Fill this in straight after the game day, while memory is fresh:

  • Did monitoring catch it, and how fast? ______

  • Did the right people get alerted? ______

  • Did the runbook actually work? ______

  • What surprised us? ______

  • Top fix to ship this week: ______

Step 8: handle the "you're going to break production?!" pushback

  • Controlled chaos is safer than waiting for the uncontrolled kind.

  • Finding weaknesses in a game day is cheaper than finding them in a real incident.

  • The team practises response under low pressure.

  • Start small, demonstrate value and build buy-in gradually.

How to use it

  • Who: the service owner, the on-call engineer and one facilitator who does not touch the keyboard. Half a day for a first game day, including the debrief.

  • Output: a written hypothesis and steady state, the scenario you ran, a completed debrief and a short list of fixes with owners.

  • Done when: the top fix has shipped and the same fault has been re-run against it, with the result written down.

Where this changes

A small team with one service can run this informally: one scenario, one afternoon, notes in a shared doc. Larger or regulated organisations need more: a standing schedule, change-management sign-off before each experiment, and evidence kept for audit. As confidence grows, the work moves from staging towards production and from manual faults towards automated, repeatable experiments.

The reasoning behind this kit is in chaos engineering for testing reliability. A game day is also the cheapest place to rehearse roles, so pair it with incident command for software teams, and if the debrief shows your runbook didn't hold, rebuild it with the incident response runbook template.

Final takeaway

A plan for failure that has never been exercised is a hope, not a plan. Pick one scenario, state what you expect, bound the blast radius and run it on a weekday afternoon. Whatever breaks is cheaper to learn about now than at 3am.

Sources

bottom of page