Find More Bugs

David MacIver @drmaciver

https://bit.ly/finding-more-bugs-pycon-uk

Your software has bugs

NASA takes bugs very seriously - the cost of bugs is very high, so they invest a lot in testing Also medical software and avionics

Most web stuff - bugs are embarrassing, but not critical

Your users should not be your QA team

Bug test cases - I did a thing, I expected this, this happened But most users don’t know how to communicate bugs Or they use bugs to make money from you

So what do you do?

Make it cheaper to find bugs Users as QA - there are many many users - testing is very parallelisable Computers are faster than people, so use them

hypothesis

inspired by QuickCheck (Haskell

from hypothesis import given
import hypothesis.strategies as st

def mean(xs):
    return sum(xs) / len(xs)

@given(st.lists(st.floats()))
def test_mean(xs):
    mean(xs)

This finds division by zero error and shows you the input that gives the error

@given(st.lists(st.floats()))
def test_mean(xs):
    m = mean(xs)
    if xs:
        assert m <= max(xs) and m >= min(xs)

Can say the empty list is not valid

@given(st.lists(st.floats()) min_size=1)

Can say the list cannot contain nan or infinity

assume(not any(math.isnan(n) or math.isinf(n) for n in xs))

More issues - slightly odd floating point edge case issues