SAP Data Quality

SAP’s flagship data quality offering is “SAP Data Intelligence Cloud,” a cloud-native product. This includes features such as data profiling, data validation, data merge, business glossary, data lineage and data governance. This product was released in 2019, an evolution of the previous SAP Data Hub product. It is based on the standard Docker-Kubernetes architecture, supports Python, Go, and JavaScript as internal engines, and can orchestrate R, TensorFlow and Spark amongst other external engines.

It is complementary to, but in effect supersedes, two previous on-premise products, SAP Data Services and SAP Information Steward. SAP Data Intelligence has bi-directional metadata exchange with these two older products, which are still fully supported, allowing their customers to migrate to SAP Data Intelligence at their own pace. By mid 2022 SAP had well over 500 customers of their new cloud-based Data Intelligence product, with 77% growth in their customer base in the previous year.

Customer Quotes

“Our customers are increasingly required to navigate a complex web of global tax policies and regulations. We need an approach to model the sophisticated corporate structures of our largest clients and deliver an end-to-end tax solution. We use a microservices architecture approach for our platforms and are beginning to leverage Amazon Neptune as a graph-based system to quickly create links within the data.”
Thomson Reuters

SAP Data Intelligence Cloud has the features that you would expect of a modern data quality solution. It supports data profiling, so examining, analyzing, reviewing and summarizing data prior to further data quality work. Data profiling is not just limited to customer data. Data can be validated against business rules, and potential duplicate records can be detected. Duplicate records can then be merged, and incomplete or inaccurate data cleansed. Location data can be validated by the use of the prebuilt integration with SAP’s Data Quality Management microservices. This deals with issues such as misspelled street names, missing postal codes etc, as well as allowing geocoding and data enrichment. In one case study, a construction company used this ability when hiring workers, with the address validation feature used for onboarding new workers.

The Data Intelligence product has a unified data catalog that curates data with centralised authorisation and security, applying data quality business rules. Metadata can be interrogated to detect the origins of data sources, and there is a business glossary so that company-specific terminology and definitions e.g. of terms like “net sales” can be applied. There is also monitoring of data quality and the scheduling of tasks, for example prompting data stewards to review possible duplicate records that require human intervention. The product has connectivity to a wide-range of data sources, both SAP and non-SAP (including relational and many non-relational databases, numerous file types and even old COBOL copybooks), and is extensible so that partner companies can build specific complementary solutions on top of the core functionality. One nice feature is that customers can examine profiled datasets and supply comments and ratings which are then visible to other authorized colleagues. The product has yet to provide the wide range of matching algorithms of its predecessor, but we expect to see that situation improve in the course of future product development.

SAP Data Intelligence is a modern, cloud-based data quality solution that has reasonably complete functionality, especially when used in combination with other SAP services such as its location validation microservice. It is well integrated with other SAP services, allowing connection to both SAP and non-SAP data, and its heritage as an independent product that was acquired by SAP means that it is by no means tied to core SAP ERP data. In practice, one customer (Evonik Industries AG) achieved significant tangible benefits by tracing errors in material classification and reducing out-of-stock situations by improving the accuracy of packaging specifications. This company halved the time for systems maintenance tasks, and speeded up the time to process complex packaging information from their suppliers by a factor of seven. Obviously, benefits will vary depending on specific use cases and industries, but SAP Data Intelligence has gained a “Top Rated” award from independent firm TrustRadius based on customer feedback, demonstrating that many companies are deriving value from the product.

The Bottom Line? 

The SAP Data Intelligence product is a natural solution for SAP customers that want to improve their data quality, which is an increasingly important issue, especially in industries that have significant regulatory requirements, but also available on-premise with good links to such as financial services and pharmaceuticals. Its cloud-native architecture but also good links to existing on-premise data and software means that customer can deploy their data quality solution in support of a migration of their core systems from on-premise to cloud at their own pace. Customers of prior on-premise SAP offerings in this area can be reassured by the bi-directional metadata exchange capabilities between this product and the older tools, easing migration at a pace that the customer can dictate. SAP’s strong market position will and indeed already has attracted plenty of partner companies to build extensions and complementary products on top of the core capabilities to round out any functionality gaps and allow application to specific industry niches.