Denodo

Fig 01 - Logical data architectures

Denodo has all the elements needed for such a data fabric architecture and has steadily developed these over its 25 years in business. To actually satisfy data queries dynamically brings with it several challenges. With data scattered among assorted systems (at least 400 source systems for a typical large enterprise, according to one survey), there is clearly a mix of source databases and applications. Some data may be tucked away in SAP or Salesforce, other data in specialist marketing or engineering systems, each of which may use different databases and file structures. Denodo has built a kind of uber-optimiser that analyses a query and attempts to satisfy that query in the most appropriate place, along with all the required modern data protection mechanisms. For example, it will try and push computationally intensive queries down to an underlying database, though it also has a dynamic MPP execution engine as a supplement, based on the Presto open-source initiative. Denodo monitors user inquiries and keeps track of which data sources are commonly accessed. All the data source access and the data delivered through Denodo are protected by the embedded policies engine supporting ABAC and RBAC.   Its machine-learning approach then suggests appropriate data aggregation and caching to improve performance over time.

Fig 02 - Real-time execution

This approach does not try to do away with master data management initiatives, where multiple versions of data about customers, products, assets etc are managed and consolidated. Instead, a master data management hub is just another source system as far as Denodo is concerned, and of course, such a source will simplify the task of the Denodo optimiser. End users see a data marketplace based on the underlying catalogue, which itself can apply business policies such as attribute level security, perhaps masking records that contain sensitive data. A workflow engine is also provided, so for example a user might request some data and discover that they need access to it; they can raise a request within the system and this request is assigned to the appropriate security administrator. Once a user has the necessary access then they can invoke a query, either directly within the product or via a third-party tool like Tableau or Qlik. There is a data lineage visual tool that shows the sources of data and for example, whether these have been through some prior transformations.

Denodo also provides a natural language query capability based on ChatGPT, whose large language model is trained on the data definition language that Denodo has access to in its catalogue. The platform monitors queries and provides scheduling capabilities, for example, to set up regular pre-aggregation of certain data that it detects is often accessed. This is a dynamic process, so the system detects new data sources, and can automatically detect changes in underlying database schemas of source systems Such changes are highlighted in the Denodo system and the catalogue can be adjusted to stay in line with the changed underlying data landscape.

I examined a large number (many dozens) of public customer testimonials of Denodo, and there is no doubt that the technology, which at first glance sounds almost too good to be true, turns out to be well-proven in practice in some very large and challenging customer environments. In this case, the data fabric story does not unravel when you pull on the threads.

The bottom line

Denodo has been a pioneer of data virtualisation and as such is in a leading position in the emerging data fabric market. It not only has the core components necessary for implementing a data fabric architecture, but these components are mature and proven in many large companies. If your company is considering implementing a data fabric architecture then Denodo is an excellent place to start.

Fig 1 - Denodo Platform

The Denodo platform performs data and metadata discovery, data lineage, data governance and change impact analysis. It has a wide range of connectors to assorted databases, files and sources, both on-premise and cloud based. It is cloud based, running on AWS, Azure, and Google Cloud.

Denodo acts as a data virtualisation layer sitting above any existing data warehouses and data lakes and other source systems, including file systems like XML or JSON. It acts as a virtual data layer, displaying data in abstracted form and providing unified data access across multiple sources and locations, with the idea of democratising visibility to data assets.

Clearly it is a non-trivial task to query across connected data sources. As well as the physical issue of dealing with distributed or federated queries across differing data sources and formats, there is the issue of meaning. What is a “product” in one system may have multiple definitions, and different versions of product data may exist across various systems and be classified in different ways. Denodo is complementary to master data management and physical data warehouses in that it is does not focus on resolving such definition inconsistencies or mapping them. Instead, it concentrates on being to run queries across multiple source systems in an efficient way. It uses machine learning to understand data usage patterns and can make recommendations, for example to suggest calculating certain aggregates, and can cache results to speed things up.  It can enforce consistently in policy definitions and also has some data quality functionality. Denodo can be regarded as complementary to master data management solutions, for example being able to use a master data management hub as a data source, if such a thing already exists.

In the latest software version, there is a natural language capability that leverages OpenAI’s ChatGPT AI, allowing customers to phrase queries in a natural language form rather than having to understand data structures. Denodo has an embedded query optimisation engine in order to resolve these queries as efficiently as possible.

In terms of data governance functions, Denodo has have role-based access control, and can enforce policies like data masking based on a user profile (e.g. for GDPR compliance) and metadata. The Denodo “Design studio” shows a logical model of the data, and the platform also allows query scheduling and monitoring.

Denodo provides support for data virtualization, allowing data to be consolidated for analytics purposes without the necessity of building a physical data warehouse. Since data warehouses are usually quite lengthy to build and tend to be brittle in the face of structural change in the underlying data sources, that is clearly a benefit. However, Denodo does not attempt to carry out semantic mapping, for example linking up multiple definitions of context data like customers or products, as would typically be dine either in a master data management or a data warehouse project (which might itself use the output of a master data management hub for its dimensions). Instead Denodo can treat an existing master data hub or data warehouse as just another data source.

This approach must have some limitations, or else nobody would have bothered with doing master data management or data warehousing in the first place. After all, resolving semantic inconsistency and duplication of data is really the hard part of a conventional data warehouse project: data about products and customers is scattered amongst systems, sometimes with inconsistent definitions, and there may be competing or overlapping versions of the same data. Different source systems may for example, have differing hierarchies of product classification depending on the application, and data quality needs to be assured: data may be incomplete, be out of data or duplicated in some source systems. Resolving those semantic and definition issues is at the heart of a master data project, and these issues cannot just be wished away by the ability to run a federated query. Nonetheless, the ability to run efficient federated queries is clearly useful in many contexts, and Denodo’s steady customer growth implies that many customers are getting value from the approach, even if it is does not make data warehouses obsolete overnight.

The Bottom Line

Denodo has established itself as a leader in the data virtualization space, allowing a growing customer base to gain some of the benefits of a data warehouse without the necessity of building a permanent data warehouse where data is physically consolidated.