Testing is broken

There are several problems with testing. The first is that outsourcers, systems integrators and so forth have little interest in the adoption of testing tools that might help to speed up or automate testing processes, because they are body shops. Better tools mean less bodies to sell. The second is that developers and testers tend to think that if they aren’t hand coding and working with a command line interface then they aren’t “real” developers: Too much “not-invented here”, and, “it may be good enough to automate business processes but we’re above that sort of thing” they claim. The third problem is that licensing testing tools is a capital expenditure and hiring bodies is an operational cost, so the bean counters are the third barrier to improvement.

The net effect of these attitudes is that there aren’t many vendors specialising in the testing market and there are even fewer who are being innovative. There are too many incentives to maintain the status quo, which plays into the hands of the body-shoppers and hand coding junkies.

Consider, for example, manual testing. This is defined by Wikipedia as “the process of manually testing software for defects. It requires a tester to play the role of an end user and use most or all features of the application to ensure correct behaviour. To ensure completeness of testing, the tester often follows a written test plan that leads them through a set of important test cases“. For example, you have to manually test every possible combination of entries that a user might make while working in a browser: what happens if you click this button followed by that one, or vice versa?

If you haven’t ever done manual testing it is easy to see how boring and time-consuming it must be. Do this, do that, note results. Do something else, note results. And so on. And on. Often for weeks or months at a time.

However, it’s not just the manual execution of tests that eats up time and budgets. Consider the test cases that manual testers are testing against. In the vast majority of companies these test cases are generated manually too. And even when that’s not the case, the majority of the tools on the market are either too complex or not sufficiently complete. For example, pairwise testing takes no account of real-world constraints.

Nor is this all. Management of test cases is, in most cases, non-existent, and old test cases never die. In addition, duplication of test cases is rife, which is another example of the not-invented here syndrome: why reuse an existing test case when you can build your own? And when it comes to actually performing testing there is, again, too much duplication of testing, over-testing of the wrong things and under-testing of the right things.

The bottom line is that testing is broken. It needs far more automation. Every other process in the business is subject to automation: why not testing?