Property-Based Testing: A Beginner's Tour

The checkout suite was green. Forty example-based tests, all passing: a $30 order with no discount, a $200 order with the bulk discount applied, a single item, an empty cart. The pricing code had shipped twice. Then someone added a four-line property test that said, in plain terms, adding one more item to an order should never make it cheaper. It failed in under a second — and handed back the smallest input that broke it: a one-cent item, quantity nine.

Nine items cost more than ten. The discount cliff had been sitting in production the whole time. No example test caught it, because no engineer thought to write the test for quantity nine next to quantity ten. That is the gap property-based testing is built to close: not the inputs you remember to check, but the class of edge-case input that example-based tests routinely miss because a human has to think of each one by hand.

This is a beginner's tour of how that works, with code you can run. If you want the ground under it first, Testing Fundamentals: Why We Test sets the frame.

Two different questions

An example-based test asks: for this input, is the output what I expect? You supply the input, you supply the expected output, and the test confirms one specific pair.

def test_reverse_twice_returns_original():
    assert reverse(reverse([1, 2, 3])) == [1, 2, 3]

That test is worth having. It documents intent and it is easy to read. But it only ever checks [1, 2, 3]. Empty lists, single elements, duplicates, negatives, a list of ten thousand — none of them are covered unless you write each one out.

A property-based test asks a broader question: is there any input for which this property is false? You describe the property; the framework manufactures the inputs and tries to break it.

from hypothesis import given, strategies as st

@given(st.lists(st.integers()))
def test_reverse_twice_returns_original(xs):
    assert reverse(reverse(xs)) == xs

Here xs is not a value you chose. Hypothesis — the standard property-based library for Python — generates it: empty lists, singletons, long lists, lists full of duplicates, lists of large and negative integers, deliberately reaching for the awkward cases. If any generated list violates reverse(reverse(xs)) == xs, the test fails and reports the offending input.

You can try this today on the simplest function in your codebase. Find something that has an obvious "and back again" shape — encode/decode, to-string/from-string, add/remove — and write the one-line property that says doing both returns you to where you started. If it passes on thousands of generated inputs, you have learned something a single example never told you.

A worked example: the discount that punished larger orders

(Developed example — composite scenario.)

Here is the pricing function from the opening, reduced to its core. The rule from the brief was "15% off orders of 10 or more."

def order_total(unit_price, quantity):
    subtotal = unit_price * quantity
    if quantity >= 10:
        return subtotal * 0.85   # 15% off orders of 10+
    return subtotal

The example-based tests read like a reasonable spec, and every one of them passes:

def test_small_order_has_no_discount():
    assert order_total(10.0, 3) == 30.0

def test_large_order_gets_15_percent_off():
    assert order_total(10.0, 20) == 170.0

def test_single_item():
    assert order_total(4.99, 1) == 4.99

Green across the board. The tester picked quantities 3, 20, and 1 — sensible representatives of "small," "large," and "edge." None of them sits on the seam where the discount switches on.

Now the property. What should be universally true of a total that no single example pins down? One candidate: a customer who buys one more unit should never pay less overall. Buying more can cost the same or more; it should not cost less. Written as a property:

from hypothesis import given, strategies as st

# prices in whole cents keeps the arithmetic exact and readable
prices = st.integers(min_value=1, max_value=100_00)
quantities = st.integers(min_value=1, max_value=100)

@given(prices, quantities)
def test_adding_an_item_never_lowers_the_total(unit_price_cents, quantity):
    unit_price = unit_price_cents / 100
    assert order_total(unit_price, quantity + 1) >= order_total(unit_price, quantity)

Run it, and Hypothesis fails almost immediately. But the first failing input it stumbles on is not the one it reports. Internally it might trip over unit_price_cents=6137, quantity=9 — a messy pair that is hard to reason about. Then it shrinks: it repeatedly simplifies the failing input, checking whether the smaller version still fails, until it cannot shrink further. What it reports is the minimal counter-example:

Falsifying example: test_adding_an_item_never_lowers_the_total(
    unit_price_cents=1, quantity=9,
)

One cent, quantity nine. Do the arithmetic it is pointing at: order_total(0.01, 9) is 0.09, and order_total(0.01, 10) is 0.01 * 10 * 0.85 = 0.085. Ten items cost less than nine. The 15% discount at the tier boundary is larger than the value of the single item that crosses it, so the total dips exactly at quantity ten.

That is a real defect with a real business consequence. A customer ordering nine units pays more than one ordering ten. The store's margin has a cliff nobody priced. And it is precisely the shape of bug example-based testing leaves uncovered: it lives between the quantities a human would think to type.

The discovery move here is worth naming, because it is where the failure actually began. The requirement — "15% off orders of 10+" — was never interrogated. Nobody asked the one intake question that would have caught this: can a larger order ever total less than a smaller one, and what is the most we can discount at the boundary before that happens? The property test is the mechanized version of that question. The fix is a scoping decision, not just a code change: cap the discounted price at the pre-threshold price, apply the discount only to units beyond the tenth, or shrink the boundary discount below the marginal item's share. Each is defensible; the point is that the property forces the conversation the spec skipped.

To try this on your own pricing or scoring code today: write the monotonicity property — "more input should never produce a smaller result" (or the reverse, whichever your domain demands) — and let the generator hunt for the seam.

Finding a property when the code seems to have none

The reverse example almost writes itself. Most real code does not announce its properties, and beginners stall here: my function just computes a total / formats an address / routes a request — what property does it even have? There are a handful of reliable moves for surfacing one.

Round-trip. If your code transforms data one way, something usually transforms it back. Parse and serialize, encode and decode, save and load. The property is that the round trip returns the original.

@given(st.dictionaries(st.text(), st.integers()))
def test_json_round_trip(data):
    assert json.loads(json.dumps(data)) == data

This one catches the unicode-key and empty-object cases you would never enumerate, and it is where round-trip properties routinely find real serialization bugs.

Idempotence. If applying an operation twice should equal applying it once — normalizing a string, deduplicating a list, canonicalizing an address — that is a property: f(f(x)) == f(x).

Invariants. Something that must stay true regardless of input: a total is never negative, a sorted list is still sorted, a balance never exceeds its cap, an account's debits and credits net to the recorded balance. The discount example used monotonicity, which is an invariant across related inputs.

Oracle — compare to a known-good implementation. This is the underrated one. When you rewrite the slow-but-correct function for speed, or replace a legacy pricing engine, you already have a reference. The property is that the new implementation agrees with the old one on every generated input:

@given(prices, quantities)
def test_new_engine_matches_legacy(price, qty):
    assert new_order_total(price, qty) == legacy_order_total(price, qty)

You do not need to know the right answer for each input — only that two implementations that are supposed to agree actually do. That single property can guard an entire refactor.

Pick one of these four for a function you are unsure about, and you will almost always find a property hiding in it. The one you should not write is the property that restates the code — asserting total == price * quantity * 0.85 just re-runs the implementation and tests nothing.

Where property-based testing pays off — and where it doesn't

It is not a universal replacement for example-based tests. Its leverage is highest where a general rule genuinely holds across all inputs, and lowest where behavior is a pile of specific, human-decided cases.

Code kind	Good property?	Why
Parser / serializer (round-trip)	Strong	parse(serialize(x)) == x holds for every valid x; generators hit the encoding edges you'd never list
Sort / algorithm (invariant)	Strong	Output stays sorted, same length, same multiset — checkable on any input without knowing the answer
UI / rendering flows	Weak	"Correct" is visual and human-judged; there's rarely a property to state, and generated interactions are noisy
Side-effect-heavy code (I/O, network, DB)	Weak	The interesting behavior is the effect, not a pure input→output rule; setup cost per generated case is high
Business rules (discounts, eligibility)	Mixed	Often no single invariant, but boundaries and monotonicity are exactly where property tests earn their keep

Read the "Mixed" row against the developed example: the discount code had no tidy invariant for its output value, yet the monotonicity property at the tier boundary found a shipped bug. When a business rule has any cross-input relationship — ordering, caps, "should agree with the old rule" — that relationship is your property. When it is a flat lookup table of unrelated cases, stay with example tests.

This is the same layering logic behind The Test Pyramid vs the Test Trophy: Choosing a Model: property-based tests are a technique you place within your unit and integration layers, not a new layer of their own. Use the table above to decide, function by function, whether a property will carry weight — and where it will, add one alongside your examples rather than replacing them.

Why the shrunk counter-example is the real gift

The failing input matters less than the minimal failing input. A framework that reported unit_price_cents=6137, quantity=9 would have technically found the bug and left you to figure out why. Shrinking does the diagnostic work: it strips the input down to unit_price_cents=1, quantity=9, which points a finger straight at the tier boundary. The smallest input that still fails is usually the clearest statement of the cause.

That minimal case is also your next example-based test. Once a property finds a bug, lock the specific counter-example in as a permanent regression test with a name that explains it — test_ten_items_never_cheaper_than_nine — so the exact defect can never return quietly. This is how property-based and example-based testing combine in a healthy suite: the property is the wide net that finds the unknown case; the example test you pin from its output is the tripwire that keeps it fixed. A well-structured suite runs both, and Unit Testing Best Practices: A Practical Guide covers how to keep that combination readable as it grows.

When a property fails, resist the urge to just widen the property to exclude the case. Read the shrunk input, decide whether the code or the spec is wrong, fix it, and capture the minimal case as an example. Do that this week on one function and you will have a regression test you could not have written by imagination alone.

Where to start

You do not adopt property-based testing across a codebase in an afternoon. You add one property to one function and let it earn trust.

1. Pick a pure function — input in, value out, no I/O. Pricing, parsing, formatting, and sorting are ideal first targets.

2. Name one thing that must always be true of it: a round trip, an idempotent operation, an invariant, or agreement with a reference implementation.

3. Write it as a single @given property and run it. If it passes on thousands of inputs, you have earned confidence no example test could give.

4. If it fails, read the shrunk counter-example, fix the code or the spec, and pin that minimal case as a named example-based regression test.

Keep each property small enough that a teammate can read it and agree it should hold. A property nobody understands is worse than no property, because a green result on it means nothing. Record the ones you keep the way you would any other case — A Test Case Template That Stays Maintainable works for properties too: state the rule, the generator, and the counter-example you pinned.

Sources

• David MacIver et al., Hypothesis documentation — generators (strategies), the @given decorator, and shrinking behavior. https://hypothesis.readthedocs.io/

• Koen Claessen and John Hughes, "QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs," ICFP 2000 — the original formulation of property-based testing and automatic shrinking, on which Hypothesis and its peers (fast-check, jqwik, proptest) are based.

No specific bug-catch percentages are cited here on purpose: figures of that kind circulate without reliable measurement behind them. The claim this article stands on is qualitative and mechanistic — property-based testing surfaces a class of edge-case bug that example-based tests miss because a human must enumerate each case by hand — and you can verify that mechanism yourself with the runnable examples above.

Keep learning. This article is part of the Software Testing Foundations path in the ShiftQuality Learning Center.