Traceability in testing

Change is a constant and that creates problems for testing environments. In particular, there is a problem with test components and particularly test scripts, especially if these are written and maintained manually. This is because the cost of manually maintaining scripts can be prohibitive. In fact, and we can generalise here – not just to testing but to any sort of development process – maintenance, especially manual maintenance, is to be avoided if at all possible. For example, we spoke not so long ago with a company that had so many ETL (extract, transform and load) scripts – tens of thousands – that the department literally had no time to do anything other than to maintain those scripts.

The question is therefore how to avoid manual maintenance or, at least, to minimise it (even in an automated environment you will need some sort of manual oversight)? The short answer, beyond saying simply “automation”, is that you need traceability from requirements, through test cases and test scripts, to the data and your expected results. And it is only if you have this traceability right through the environment that you can successfully expect to implement automation that will take away many of those expensive manual processes.

What does that mean in practice? It means that when a requirement is changed then relevant amendments are automatically generated (or retrieved if you have appropriate test cases in your library) for all the subsequent steps in the testing process: the test cases, the scripts, the data that needs to be run through the tests, and the results that you expect from those tests.

Achieving this is not as simple as stating it. In reality you are going to need an integrated suite of tools that starts with requirements capturing and test case and test script generation, combined with test data capabilities. In this latter case you will want test data management for in-house data but will need to integrate with service virtualisation for third party data or other data that is not easily accessible. This suggests that point products will not be suitable as these will only resolve, at best, a part of the problem. As an aside, and taking a broader perspective – from requirements through development to testing and provisioning – then we are talking about an integrated suite of products that combine to provide continuous delivery.

Going back to the testing environment, if we are assuming that traceability is implemented throughout, then everything depends on the original requirements, or changes thereto. This in turn means that requirements need to be captured in a formal manner in some sort of model (where the word “model” is used here in its most abstract sense) so that when you make a change to the model then everything else is automatically updated by virtue of the traceability back to the model. Note that this doesn’t necessarily mean generating new test cases, it may mean recognising that there is an existing test case that can be reused to support the current scenario.

What is required is joined-up thinking or, more accurately, joined-up product solutions. This should include test case management (managing reusable testing assets) as well as the other capabilities discussed.