Testing Eventual Consistency: What "Eventually" Has to Mean
Composite scenario, built from common failure patterns; not a real organisation. A regional retailer's order service wrote each order to its database and published an OrderPlaced event. A consumer read the event and updated the read model behind the customer's "My orders" page. The team had a test for it, and it was the right shape: place an order, then wait up to five seconds for it to appear in "My orders". No fixed sleep, a deadline, a clear failure message. It passed in CI for a year.
During a holiday sale the consumer fell behind. Lag on the event topic climbed to about forty seconds. Customers placed an order, landed on "My orders", saw nothing, and did what anyone would: they placed it again. Support spent the following week cancelling duplicates and refunding double charges.
The test had been correct. The order did appear eventually. What it never asked was how eventual was acceptable, who was looking during the gap, and what they would do while the data was stale.
This is a companion to Testing the Hard Parts: Async, External Dependencies, and State, which covers awaiting async results, polling instead of sleeping, and testing concurrent access. Here the focus is narrower: systems where a write succeeds now and becomes visible elsewhere later, and what a test has to assert about the "later".
Why "eventually" passes in CI and fails in production
Eventual consistency, as Werner Vogels described it in 2008, guarantees that if no new updates arrive, all reads will eventually return the last written value. It says nothing about how long that takes. Peter Bailis and colleagues measured this for Dynamo-style stores in 2012 and found that in the average case such systems often return consistent data within tens of milliseconds, while in the worst case staleness is unbounded.
That gap between the average and the worst case is why these tests mislead. CI runs with an idle broker, one consumer and no backlog, which is the average case at its best. Production under load is where the tail lives. A test that waits five seconds in CI is measuring the quiet case and calling it a guarantee.
It is also why these tests are a leading source of flakiness. In Luo and colleagues' 2014 study of flaky tests in open-source projects, the largest root-cause category was async wait: 74 of the 161 fixes they classified, or 45%. About a third of those tests had used a fixed time delay to enforce an ordering.
The discovery move. Before writing the test, ask the product owner one question: after a user does X, how long until Y can see it, and what happens if someone acts on the old data in between? The answer is a requirement, a staleness budget, and almost nobody writes it down. The retailer's five seconds came from a developer's guess, not from anyone deciding that five seconds of invisible orders was acceptable.
The test you can run: find every eventually, waitFor or polling helper in your suite and list its timeout. For each one, find the document where someone agreed that number. If none exists, the test enforces a guess.
Assert the budget, and measure the lag
Once there is a staleness budget, the test can do two jobs: check convergence within the budget, and record how close it came.
// Staleness budget agreed with product: 2s at p99 for "My orders"
const STALENESS_BUDGET_MS = 2000;
test('order is visible in My orders within the staleness budget', async () => {
const placedAt = Date.now();
const { orderId } = await placeOrder(customer, cart);
await eventually(async () => (await myOrders(customer)).some(o => o.id === orderId),
{ timeout: STALENESS_BUDGET_MS * 3 }); // diagnose, don't hang
const lag = Date.now() - placedAt;
recordMetric('my_orders_visibility_lag_ms', lag); // trend this in CI
expect(lag).toBeLessThan(STALENESS_BUDGET_MS);
});
The polling timeout is deliberately longer than the budget. If the order takes three seconds, you want a failure that says "converged in 3,100 ms, budget 2,000", not "timed out", because the first tells you the system works but has drifted, and the second tells you nothing.
The same number belongs in production monitoring: consumer lag on the topic, alerting well before it reaches the budget. A budget that is tested but not monitored only protects you on days that look like CI.
The test you can run: plot the recorded lag from your last 50 CI runs. If the trend is rising, you will breach the budget before the test ever fails.
Test the gap, not just the convergence
The retailer's failure happened inside the gap, so that's where the most valuable tests go. They need the gap to exist on purpose, which means injecting lag.
test('a customer who resubmits during projection lag gets one order', async () => {
await consumer.pause(); // simulate 40s of lag
const key = idempotencyKey(customer, cart);
const first = await placeOrder(customer, cart, { idempotencyKey: key });
expect((await myOrders(customer)).length).toBe(0); // the stale view the customer saw
const second = await placeOrder(customer, cart, { idempotencyKey: key }); // natural retry
expect(second.orderId).toBe(first.orderId);
await consumer.resume();
await eventually(async () => (await myOrders(customer)).length === 1);
expect(await chargesFor(customer)).toHaveLength(1);
});
Two fixes made that test pass at the retailer, and they answer different questions.
Read-your-writes for the person who wrote. Vogels' paper lists read-your-writes as one of the practical variations on eventual consistency: a process that wrote a value always sees its own write afterwards. The retailer made it true for the customer by showing the order from the write model on the confirmation page, and by merging the session's just-placed orders into "My orders" until the projection caught up. Other customers and the warehouse could still see stale data. The customer who had just ordered could not.
Idempotency for the retry nobody can prevent. Even with read-your-writes, a customer on a flaky connection will press the button twice. An idempotency key generated on the client and checked on the write path turns the second submission into the first.
The test you can run: pause the consumer behind your most customer-visible read model in a test environment, then perform the natural user action, whether that's a refresh, a resubmit or a back-button retry. Count the side effects. More than one is a bug your convergence test can't see.
Duplicates and reordering are the normal case
Most brokers deliver at least once, so consumers see duplicates after a crash or rebalance, and events for the same entity can arrive out of order across partitions or retries. A projection that is right only when every event arrives once and in order will be wrong in production.
The test for this is cheap and catches a lot: build the projection from the event history, then rebuild it from the same history with every event delivered twice and with independent events shuffled. The three results should be identical.
test('projection is idempotent and order-tolerant', () => {
const events = fixtureHistory(); // 200 recorded events
const once = project(events);
const twice = project(events.flatMap(e => [e, e]));
const shuffled = project(shuffleWithinKey(events)); // preserve per-order sequence
expect(twice).toEqual(once);
expect(shuffled).toEqual(once);
});
If shuffled differs, decide whether your consumer should reject out-of-order events (using a per-entity version number) or tolerate them by design. Either is defensible. Leaving it to whatever order the broker happens to deliver isn't.
The dual write: where events go missing
The retailer's order service committed to the database and then published the event. If the process dies between those two steps, the order exists and the event never does, and the read model never converges, which no amount of waiting fixes.
The standard remedy is the transactional outbox pattern, documented by Chris Richardson: write the event to an outbox table in the same transaction as the business change, and have a separate relay publish it. To test that the remedy is actually in place, test the crash.
The test you can run: in an integration environment, inject a failure immediately after the database commit and before the publish. Then check whether the event is eventually delivered. If it isn't, you have a dual write. Schedule a reconciliation job comparing source-of-truth counts to read-model counts as a backstop either way.
Consistency contracts change under you
Consistency guarantees belong to dependencies, and dependencies change. Amazon S3 was eventually consistent for some operations for most of its history. On 1 December 2020, AWS announced strong read-after-write consistency for all S3 requests at no extra cost. Code written for the old model carried retries and delays that became dead weight overnight. The reverse is more dangerous: a new region, cache layer or vendor that weakens a guarantee your tests assumed.
The test you can run: for each data store, cache and queue your service reads from, write down the consistency guarantee you rely on, with a link to the vendor's documentation. Review the list whenever a dependency is upgraded or replaced.
Trade-offs
Approach | Gains | Costs | Best suited for | Warning signs |
Poll with a deadline only | Simple, cheap | Proves convergence on a quiet day, nothing more | Low-stakes internal views | Timeouts nobody agreed |
Staleness budget plus lag metric | Turns "eventually" into a requirement; early drift warning | Needs a product decision and monitoring | Customer-visible read models | Budget tested but not alerted on |
Lag injection plus user-action tests | Catches harm done inside the gap | Test harness must control consumers | Flows where users act on what they see | Duplicate orders, double charges |
Duplicate and shuffle replay | Cheap; finds non-idempotent projections | Needs recorded event fixtures | Every event-sourced projection | Projection differs on rebuild |
History checkers (Jepsen, Elle) | Find isolation anomalies in the data store itself | Specialist effort | Choosing or upgrading a database | Vendor claims nobody has verified |
For that last row, Kingsbury and Alvaro's Elle checker (2020) infers isolation anomalies from recorded transaction histories. It found unexpected anomalies in each of the four databases they tested. Most teams don't need to run it, but it's a reminder that a store's documented guarantee is a claim to verify, not a fact to assume.
When strong consistency is the right answer
Sometimes the right test result is "make it consistent". If the staleness budget the product needs is effectively zero (a balance before a withdrawal, stock before the last unit sells), you are looking for a reason to read from the system of record on that path rather than test around a projection. Eventual consistency buys availability and latency. It is worth paying for only where users can tolerate stale data.
A method for one read model
Agree the budget. Ask the product owner the discovery question and record the answer, for example "2s at p99, submitting user sees their own write immediately".
Add the convergence-plus-metric test with a diagnostic timeout of three times the budget.
Add one lag-injection test for the most natural user action during the gap.
Add the duplicate-and-shuffle replay test to the projection's unit suite.
Test the crash between commit and publish once, and keep the reconciliation job.
Monitor consumer lag in production with an alert below the budget.
You're done when each step has a passing test or a dashboard, and the budget is written somewhere a new team member would find it.
What to do next
Pick the read model your customers look at most. Find its eventually timeout and ask who agreed that number. If nobody did, take the discovery question to the product owner this week.
Final takeaway
A test that waits for convergence proves the system gets there eventually. It doesn't prove how long that takes or what users do in the meantime. Write the staleness budget down, test against it, and test what people do inside the gap, because that's where the damage happens.
For flaky waits in general, see Flaky Tests: Diagnosis and Cure. To keep producers and consumers from drifting apart on event shape, see Contract Testing: Keeping Services Honest, and when lag turns into interleaving bugs, work through Tutorial 7: Debug Race Conditions.
Sources
Vogels, W. (2008). "Eventually Consistent." ACM Queue 6(6), 14–19; also Communications of the ACM 52(1), 2009. cs.princeton.edu copy
Bailis, P., Venkataraman, S., Franklin, M. J., Hellerstein, J. M. and Stoica, I. (2012). "Probabilistically Bounded Staleness for Practical Partial Quorums." PVLDB 5(8). vldb.org
Luo, Q., Hariri, F., Eloussi, L. and Marinov, D. (2014). "An Empirical Analysis of Flaky Tests." FSE 2014 (FSE 2024 Test of Time award).
Kingsbury, K. and Alvaro, P. (2020). "Elle: Inferring Isolation Anomalies from Experimental Observations." PVLDB 14(3). arxiv.org/abs/2003.10554
Richardson, C. "Pattern: Transactional outbox." microservices.io
AWS (2020, 1 December). "Amazon S3 now delivers strong read-after-write consistency automatically for all applications." aws.amazon.com
