Enov8

Fig 3 - Test Data Manager profiling execution report

Data profiling in Test Data Manager (for finding your sensitive data, among other things) consists of creating and running data profiling configurations that specify the selection of data fields, and corresponding data patterns, you want to identify. You can flag fields as sensitive within these configurations, and even specify what kind of sensitivity (such as primary PII, secondary PII, or PHI) each field falls under. In addition, various preset profiling patterns and configurations are available, catering to common, international use cases. Data fields are identified using column name and/or pattern matching, and the results of data profiling are presented within profiling reports available for each profiling job individually and for all of them collectively (see Figure 3 for an example of the latter).

Profiling configurations can also contain recommended data masking (or other anonymisation) methods for each (sensitive) field. During the anonymisation process, you can either rely on these preconfigured methods or choose your own. Enov8 generally recommends using a hashing algorithm to replace sensitive values with random ones drawn from a pre-generated many-to-one lookup table. This method is flexible, non-reversible, maintains referential integrity and can produce production-like data. A selection of look-up tables for common fields are provided out of the box to facilitate this. The product also uses essentially the same method to create synthetic data sets, except that instead of anonymising a series of existing fields it simply creates a set number of them.

Other anonymisation methods are also available, and include encryption, complex ciphering, random or fixed variance (for numerical fields) and fixed value replacement (which is to say, everything is masked to the same thing). You can also implement your own masking algorithms as Python functions, with templates for the above methods provided. Similarly, you can use Python to extend or modify any of the built-in masking methods. You can also use parameters to perform calculations and leverage their results during the masking process, such as when calculating a checksum or masking a field that needs a concatenation of other masked fields.

Fig 4 - Test Data Manager executive insights dashboard

Masking can be executed manually or scheduled as part of the company’s Orchestration Manager (and in fact can be positioned as part of an automated pipeline, along with profiling and various other tasks, using the same). The masking process can be paused and resumed intelligently, so that existing progress is not lost, and ultimately produces a report detailing which masking jobs succeeded and which (if any) failed. In the latter case, error details are provided, and you can choose to rerun only the failed jobs once those errors have been fixed. Delta analysis is available for change management purposes where an update to the database schema may have introduced new fields, and masking validation available as part of the report will tell you both how production-like each field’s masked data is (using distribution analysis) and what percentage of values have been masked within each field.

Finally, for Test Data Mining, Test Data Manager offers a data library that allows you to both view and reserve data. Various data dashboards are also available (an example of which is shown in Figure 4) that can be customised both for your organisation as a whole and by each user individually.

Test Data Manager provides considerable breadth of test data management functionality. To wit, it offers a complete test data solution, providing sensitive data discovery (via data profiling), data anonymisation (via masking), and database virtualisation (when used alongside vME, which is highly recommended). It also benefits from being part of a product suite alongside Enov8’s other offerings, such as Environment Manager, Release Manager, and Orchestration Manager. In short, it allows you to protect your sensitive test data – and comply with any relevant regulations – while accelerating your test data processes and ensuring they do not become a bottleneck for your testing as a whole. Synthetic data generation is also available, although in this case we would say it is best considered as an ancillary capability for complimenting your other test data with new data that may not exist in production.

Using distribution analysis to determine how closely your masked data resembles your production data is a rarely seen, but welcome, feature. It allows you to create test data sets that are highly secure but retain the statistical properties of the original data, enabling you to test as if you were using the original data but without the accompanying privacy concerns. The selection of masking and matching methods provided by Test Data Manager are relatively minimal, but its primary method of masking – using many-to-one lookup tables – compensates for this through sheer flexibility, and you can add to the available masking methods yourself if required.

It is also worth noting that Test Data Manager is quite easy to use through its web interface, to the extent that, at least in principle, it can be leveraged by both technical and non-technical users. At minimum, this makes your testers’ lives easier. More significantly, this can allow parts of your organisation outside of your dedicated testers to assume some responsibility for test data, helping you to spread the load and utilise tribal knowledge. In theory, you could even federate out your test data management entirely, with your dedicated testers primarily acting in a test data governance capacity.

The bottom line

Enov8 Test Data Manager is a robust solution for data profiling and data masking, with additional capabilities for test data provisioning and synthetic data generation (among other things). When combined with vME, the database virtualisation offering also from Enov8, it offers a holistic and comprehensive solution for test data management, ensuring test data security, compliance, and efficiency, especially within DevOps environments.

Figure 2 – vME web interface

vME provides a dashboard user interface (shown in Figure 2) for controlling four kinds of virtualised data assets: pools, sources, snapshots, and clones. These are hierarchical: sources are derived from pools, snapshots from sources, and clones from snapshots. The process starts by ingesting a backup file for the database from which you wish to create these assets, and this can be actioned within the dashboard, as can the distribution of clones to your users.

Pools are collections of virtual storage devices that have been grouped together, and are standard technology for ZFS, the underlying system on which vME is built. Sources are essentially ingestion points for a given pool, and are usually containerised when the type of database you are working with permits it. Snapshots are point-in-time bookmarks for a given source, while clones are virtual copies of the source at that point in time. The reason clones are so small is that they only store differences between them and the snapshot from which they were created, which will initially be nothing and is unlikely to grow that much due to their distributed and often temporary nature. Clones can be either container or network mounted, and can themselves be bookmarked (effectively creating different versions of the same clone, which can then be reverted to at a later date).

Data discovery and masking is typically actioned at the source level, before a snapshot is taken, whether through Test Data Manager or a third-party product. The effects of these processes are then carried forward, meaning that you only need to find and protect the sensitive data in a given source once (at least until the source is updated) and the sensitive data in every clone created from that source will also be protected.

vME is designed to be DevOps-first. Accordingly, its functionality can be accessed via APIs or a CLI (Command Line Interface) in addition to the web interface already described. This allows it to fit neatly into existing workflows and toolchains, particularly CI/CD pipelines. The web interface also provides a file explorer for examining and manipulating containerised data. Moreover, vME’s underlying engine interacts with data at the file system block level, which means that vME is both database and file type agnostic. It also uses a federated approach, promoting a load-balanced architecture and allowing you to distribute vME application instances to teams or individuals as needed.

Finally, vME recently added the ability to leverage IBM ZD&T (the IBM Z Development and Test Environment) to create clones of z/OS mainframes. This is useful from a testing standpoint, but also for training purposes: by distributing mainframe clones, you can provide sandboxed environments to trainee mainframe users. This is helpful because there is a significant shortage of mainframe developers, and because it is normally quite difficult to give prospective developers a mainframe environment to safely play in. By addressing the latter, vME can help to address the former as well.

Database virtualisation is still an emerging technology in test data management, but its efficacy and desirability have been clear for some time. One of its major issues has been cost, with one of the most prominent virtualisation vendors exclusively targeting only the wealthiest companies, and setting its prices accordingly. This has meant that database virtualisation has, historically, been largely out of reach of both small and medium-sized enterprises. vME is one of a new breed of database virtualisation products that is available at a price point that is affordable for these vendors.

Indeed, vME’s affordable price point, easy deployment, and self-service approach give it a pleasingly short time to value. In particular, this makes it an excellent candidate for deploying to specific teams, tribes, or even individual users within an organisation, either on a permanent basis or as part of a gradual, enterprise-wide rollout. Moreover, its integration with third-party tools and DevOps-compatible design means that it will fit cleanly into many existing test environments and toolchains. Its ability to help train mainframe developers by providing z/OS clones is also notable.

The bottom line

Enov8 VirtualizeMe is a capable and compact database virtualisation solution that is oriented towards test data management. In that respect, it caters to a wide variety of enterprises and environments, whether in combination with Enov8 Test Data Manager or by integrating with existing test data management tooling. If you do not already have access to database virtualisation (and perhaps even if you do) you should seriously consider adding VirtualizeMe to your testing environment.