I played a lot of volleyball in a bygone life 🙂 and subsequently ruined my knees to the extent that I needed surgery. I got a shock when the surgeon (after a series of x-rays and checks) said to me: “Of course, we’ll only know once we’re in there”.
So here’s a body part (a knee) that’s had hundreds of thousands of years to evolve, so you’d expect that knees are pretty much the same world wide, yet an experienced and qualified surgeon puts the “we cant be 100% sure” caveat before chopping me open.
I wish we could apply the same process to testing of IT software. I remember reading about the mantra of “its harder to fix bugs once they’re in production” and that’s certainly true. However, somewhere along the way, that became the justification for test cycles being incredibly long and mind-bogglingly detailed. If my Finance software can’t balance the books, then yes, that’s a big drama. But if the “Save” button is a shade of blue that didn’t match the design screen shots – is it really worth holding back the production implementation? There are two problems with a mantra of “we will find every possible defect”:
1) You can find defects in software almost ad infinitum. They just get less and less severe, and your testing cycle bottlenecks you entire IT department.
2) You create a false confidence in the testing. “Hell, if we spent 12 months in testing, then we’ll never find any bugs once we go to Production.”
So I say – Why not take the surgeons approach? Sure, we’ll test the product to a reasonable level of satisfaction, but we’ll readily accept the fact that the ultimate test ground is only Production. “We’ll only know when its in there”.