Denodo and data governance

Denodo is a data virtualization platform. It can be thought of as a virtual data warehouse, a layer sitting in between source systems and end users, allowing queries to run across multiple source systems without the need to physically bulk copy data from the sources into a conventional, permanent data warehouse. Instead, the Denodo software provides a semantic layer that allows queries to be formulated, and then executed against the original sources in a federated fashion.

Denodo also provides data governance capabilities, allowing data policies to be defined and enforced, and providing capabilities for data lineage tracking and metadata management. There are connectors to a wide range of sources including SQL databases, XML files and JSON files.

Customer Quotes

“The Denodo Platform exposes data from sources, applies business rules for quality monitoring, and creates a proper interface that can be consumed by multiple downstream users.”
Chief Architect and Chief Data Officer, CIT Group

“Denodo helps us reduce the overall size of the data sets that need to be moved over the network. It also helps us create a logical data fabric that makes our data assets reusable and can be consumed by users spread across the globe.”
 Joshua Fletcher, Superintendent HSE Reporting at BHP

The Denodo platform performs data and metadata discovery, data lineage, data governance and change impact analysis. It has a wide range of connectors to assorted databases, files and sources, both on-premise and cloud based. It is cloud based, running on AWS, Azure, and Google Cloud.

Denodo acts as a data virtualisation layer sitting above any existing data warehouses and data lakes and other source systems, including file systems like XML or JSON. It acts as a virtual data layer, displaying data in abstracted form and providing unified data access across multiple sources and locations, with the idea of democratising visibility to data assets.

Clearly it is a non-trivial task to query across connected data sources. As well as the physical issue of dealing with distributed or federated queries across differing data sources and formats, there is the issue of meaning. What is a “product” in one system may have multiple definitions, and different versions of product data may exist across various systems and be classified in different ways. Denodo is complementary to master data management and physical data warehouses in that it is does not focus on resolving such definition inconsistencies or mapping them. Instead, it concentrates on being to run queries across multiple source systems in an efficient way. It uses machine learning to understand data usage patterns and can make recommendations, for example to suggest calculating certain aggregates, and can cache results to speed things up.  It can enforce consistently in policy definitions and also has some data quality functionality. Denodo can be regarded as complementary to master data management solutions, for example being able to use a master data management hub as a data source, if such a thing already exists.

In the latest software version, there is a natural language capability that leverages OpenAI’s ChatGPT AI, allowing customers to phrase queries in a natural language form rather than having to understand data structures. Denodo has an embedded query optimisation engine in order to resolve these queries as efficiently as possible.

In terms of data governance functions, Denodo has have role-based access control, and can enforce policies like data masking based on a user profile (e.g. for GDPR compliance) and metadata. The Denodo “Design studio” shows a logical model of the data, and the platform also allows query scheduling and monitoring.

Denodo provides support for data virtualization, allowing data to be consolidated for analytics purposes without the necessity of building a physical data warehouse. Since data warehouses are usually quite lengthy to build and tend to be brittle in the face of structural change in the underlying data sources, that is clearly a benefit. However, Denodo does not attempt to carry out semantic mapping, for example linking up multiple definitions of context data like customers or products, as would typically be dine either in a master data management or a data warehouse project (which might itself use the output of a master data management hub for its dimensions). Instead Denodo can treat an existing master data hub or data warehouse as just another data source.

This approach must have some limitations, or else nobody would have bothered with doing master data management or data warehousing in the first place. After all, resolving semantic inconsistency and duplication of data is really the hard part of a conventional data warehouse project: data about products and customers is scattered amongst systems, sometimes with inconsistent definitions, and there may be competing or overlapping versions of the same data. Different source systems may for example, have differing hierarchies of product classification depending on the application, and data quality needs to be assured: data may be incomplete, be out of data or duplicated in some source systems. Resolving those semantic and definition issues is at the heart of a master data project, and these issues cannot just be wished away by the ability to run a federated query. Nonetheless, the ability to run efficient federated queries is clearly useful in many contexts, and Denodo’s steady customer growth implies that many customers are getting value from the approach, even if it is does not make data warehouses obsolete overnight.

The Bottom Line

Denodo has established itself as a leader in the data virtualization space, allowing a growing customer base to gain some of the benefits of a data warehouse without the necessity of building a permanent data warehouse where data is physically consolidated.