A storm about testing

ISO 29119 is a set of standards for software testing. It’s not complete yet. The first three standards: “concepts and definitions”, “test processes” and “test documentation” are already in place but “test techniques” and “keyword driven testing” are expected to be published late this year and next year respectively.

So far, so good: testing needs some rigour. I don’t think anybody disagrees about that. Why then is there a petition to stop ISO 29119? And why are people blogging and tweeting against it? The first answer to those questions is that it costs around £120 to download the relevant documents, which people clearly resent. But there are more serious concerns. In particular, the final piece of the puzzle: “keyword driven testing” appears to suggest—remember that this has not been published yet, though an outline is available—that the use of keywords will be mandated to describe test cases. In other words, this looks very prescriptive. Moreover, the standard eschews alternative approaches and it suggests that there is only one right way of doing things. In effect, ISO is taking a fundamentalist stance and this, in my view, is limiting, not merely excluding other existing approaches but potentially making it harder for new and innovative methods to be introduced in the future.

The anti-camp have a further concern. This is that IS0 29119 becomes widely accepted as the industry standard for testing, required by companies both for their internal development efforts and for outsourced developments. If ISO 29119 starts to become mandated by organisations this a) places a financial burden on suppliers who have to get certified (which will drive up costs, even if more efficient testing means the opposite) and b) makes life difficult for providers who actually disagree with the standard’s provisions, either for the reasons discussed above or otherwise. And then there is the general problem with standards that they all too easily become box ticking exercises: you spend a bunch of (expensive) time documenting everything so that you can prove that you are meeting the standard rather than, in this case, actually spending your time on what you ought to be doing, which is testing.

This is actually a difficult circle to square: testing certainly needs more rigour. On the other hand, the protesters against ISO 29119 are probably those who need standards the least precisely because they already take a professional approach. The question is whether the needs of the many outweigh the needs of the few: tricky.