Hitachi Vantara Lumada

The Lumada DataOps Suite from Hitachi Vantara has five components, as illustrated in Figure 1. Also available from the company is Pentaho Enterprise Edition (currently in version 9.1) and some of the capability offered by Lumada is based on leveraging Pentaho’s (open source) technology. This is especially true of the Data Integration and Enterprise Analytics components shown in this diagram. It is less true of the other elements of the Suite, particularly the Data Catalog (based on the acquisition of Waterline Data). While this has not yet been fully integrated with Pentaho, substantial elements of this integration (for example, registering objects into the catalogue from Pentaho and writing metadata, as well as reading capabilities) are already in place.

Fig 01 – The Lumada DataOps Suite

The other important point to note about Figure 1 is the support for Lidar and other devices that are not usually supported by data integration vendors. This is indicative of Hitachi Vantara’s industrial focus and the company specifically targets manufacturing, logistics and Internet of Things applications as well as more traditional industries such as financial services, telecommunications, and healthcare.

Customer Quotes

“The functionality that Pentaho gives over proprietary vendors provides huge cost savings.”
Brussels Airport

“CERN’s systems need to manage high volumes of confidential information on its employees and their families, so security, data governance, and data integrity are all paramount. After a review of five different proprietary and open source platforms, Pentaho emerged as best adapted to our needs.”
CERN

“Hitachi Vantara has shown that it’s a true partner to customers. Pentaho is a great tool that’s evolved to meet the challenges of real people.”
Bell Canada

Fig 02 – The functionality of the Lumada DataOps Suite

The Lumada DataOps Suite aims to provide the functionality illustrated in Figure 2. That is, it is intended to support everything from the ingestion of data through to its analysis and presentation in dashboards. That said, while it includes some capabilities for data profiling within its data cataloging product (see below) it does not offer data quality or governance capabilities, for which it relies upon partners such as Collibra, as it does with data preparation   and data masking. In addition, Lumada Data Optimizer for Hadoop offers orthogonal capability by providing intelligent data tiering for HDFS environments. The company is a partner of Cloudera’s.

Fig 03 – Lumada Data Integration capabilities

As far as data integration is concerned, Figure 3 provides a snapshot of the capabilities offered though what it doesn’t tell you is that there is a distinction between Pentaho and Lumada Data Integration, where the latter consists of Pentaho Data Integration combined with Dataflow Studio, which is a web and microservices-based  application that enables scalable execution of dataflow jobs for business users rather than only being targeted at IT folk. It also doesn’t tell you that Lumada Edge Intelligence (IoT Edge) provides a lightweight environment that supports the hosting of AI and machine learning in edge devices, along with real-time KPI-based dashboarding, alerting, and support for streaming data. More generally, streaming is supported in all its varieties: micro-batches or real-time, continuous or event-driven, as you require. Also not mentioned is the fact that the company offers more than 200 connectors; support for Jupyter Notebooks along with R, Python and TensorFlow; and an extensible framework that includes SDKs, APIs, and plug-ins. And that change data capture (CDC) is supported.

Finally, we should discuss the Lumada (previously Waterline) Data Catalog. This is targeted at both enterprise data lakes and traditional data environments. It provides a complete solution for data discovery, cataloguing and compliance on these platforms, and is particularly notable for its discovery process, which is based around using machine-learning driven “data fingerprinting” to tag data consistently and intelligently using   a bespoke, machine learning driven solution for data discovery and tagging.

The discovery and tagging solution works by creating a “fingerprint” for each of your data fields. Each fingerprint is a collection of metadata – in some cases, more than 100 pieces of metadata – which includes information about both the content of your data and the context it exists in. Lumada Data Catalog can then intelligently and automatically classify and tag each of your fields based on its fingerprint. Once your data has been classified, it is exposed to your users as part of the data catalogue and is open to curation and crowdsourcing, the former of which feeds the machine learning algorithm, thus improving its accuracy over time. Different confidence levels can be attached to tags based on the likelihood that a particular tag applies.

The biggest differentiator for Hitachi Vantara’s Lumada DataOps Suite is its end-to-end capabilities from federated edge functions through to multi-cloud centralised operations. Other data integration vendors do not typically support sensors and other devices to the extent that Hitachi Vantara does. Conversely, the suppliers that focus on IoT and similar environments do not generally offer strong support for data integration in the traditional sense of that term. We should add that the acquisition of Waterline Data in March 2020 adds significantly to the capabilities that Hitachi Vantara can offer and we would not be surprised – indeed we would recommend – to see further relevant acquisitions in the future.

The Bottom Line

With Lumada being modular, and the fact that Hitachi Vantara can leverage partners when it needs to (depending on the use case), we regard the DataOps Suite as a platform play. We particularly like it within the context of distributed deployments where it can leverage its major differentiators.