Ab Initio Metadata Hub

The Ab Initio Metadata Hub acts as the data governance component for Ab Initio’s data management platform. It can be used as either a system of record or a system of reference, is able to govern technical, business, and even logical assets, and provides both business and technical lineage. It also offers data quality and reference data functionality, as well as role and responsibility management. Moreover, the Metadata Hub is closely integrated with Ab Initio’s other solutions, such as Semantic Discovery, which each provide significant additional capabilities.

The Metadata Hub separates your data assets into a number of categories. For data governance, the most important are business assets, technical assets, and logical assets, as well as reference data. In effect, these consist of business information surrounding your data (business assets), the physical reality of your data (technical assets), and logical data models that describe your data (logical assets). Each type of asset can be browsed through at your leisure, and this acts as the first of two primary means to access (and govern) your assets. The second is through one of the two lineage views that the product provides.

Both of these lineage views visualise the movement and impact of data and data assets through your system as a flowchart, and allow you to access your data assets directly by drilling down into them. Where they differ is in the perspective they take on this movement. Business lineage (shown in Figure 1) approaches it with a high-level, logical view, and illustrates how your business assets interact with and are processed by your system from a business perspective. In other words, it focuses on the interactions that matter to the business, without much regard for the physical reality. An initial view of your business lineage can be generated automatically from the relationships between your fields and your business terms (see below), but some manual effort will usually be required to make it suitable for consumption by the business.

Fig 01 – Business lineage with data quality overlay in Metadata Hub

Technical lineage, on other hand, takes the exact opposite approach, by examining how your technical, physical assets literally move through your system, how your files, fields and tables interact, and so on. What’s more, technical lineage is generated automatically from your metadata, and said metadata can be imported from a wide variety of third-party systems.

In fact, a sizable range of extractors are provided for importing metadata into the product, both for generating technical lineage and more generally. This includes support for a number of third-party products, including some direct competitors in the data governance space. It’s also possible to write your own extractors using the open documentation that Ab Initio provides. Extractors can be run either through Ab Initio’s UI or via the command line, and can therefore be scheduled by utilising the latter.

Fig 02 – Editing a business term in Metadata Hub

The product’s business glossary, as seen in Figure 2, allows you to centrally define and manage your business terms. Using the business glossary, they can be given a number of (configurable) types, such as ‘critical element’, and can be equipped with classifications such as PII (with the latter tying into Ab Initio Semantic Discovery, the company’s sensitive data discovery solution). Your terms can be hierarchical or otherwise related to other terms, as well as your physical data (again accelerated by Semantic Discovery) and other assets. These relationships are also available as a visualisation, which is generated automatically. Role and responsibility management is configurable for each term individually (and can have their own hierarchies), and a configurable workflow approval process is used to facilitate this. Search access to each term is also provided.

Your business terms can also be used to measure your data quality from a business perspective. Data quality rules can be created and added to your business terms, which will then contribute to an associated, user-configurable quality metric (typically examples include ‘accuracy’ and ‘consistency’). Each rule provides a historical performance summary as well as a list of associate terms, assets, and so on. Configurable thresholds drive data quality warnings, email notifications, or other actions if your quality metrics fall too far. Data quality checking can be run manually or automatically via scheduling. Notably, data quality information can be overlaid onto your business lineage to form a data quality heat map, allowing you to visually understand the health of your system. This can be seen in Figure 1.

The Metadata Hub offers a number of advantages as a platform for data governance. For instance, it positions lineage prominently as a part of its solution, and as you might therefore expect, its lineage capabilities are a significant draw. In particular, explicitly providing both business and technical lineage can prove very useful by allowing all of your users to easily comprehend and get what they need out of your lineage. The data quality heat map is also a notable feature. It’s unfortunate that preparing your business lineage for consumption is likely to take manual effort, but on the other hand, generating your technical lineage is completely automated, and can be accomplished using a selection of third-party – and even competitor – products, to boot.

Ultimately, though, the greatest strength of the Metadata Hub is not part of the product itself. Rather, it’s the product’s place in Ab Initio’s entire milieu that makes it so powerful. For instance, it will readily and closely integrate with Semantic Discovery, and hence add full-fledged data discovery and classification (and thereby sensitive data discovery and GDPR compliance) to your governance solution. What’s more, Semantic Discovery is far from the only integration available: Ab Initio offers a wide range of data management solutions, and in many ways the Metadata Hub acts first and foremost to bring those solutions together.

The Bottom Line

Ab Initio is a broad and highly regarded platform for managing your data. The Metadata Hub, as part of that platform, is an excellent way to bring different elements of it together in aid of data governance.