Data Preparation (self-service)

This page has been archived and merged, please visit the Data Discovery and Catalogues or Trust in Data pages for new content.

There are various functions you may need to perform in order to get data ready for analysis. You may need to cleanse data, you may need to remove or complete empty fields, you may need to de-duplicate data. You will commonly need to join data from diverse sources, which will mean identifying a join key (some common information, such as an email address) and possibly transforming the data so that it is in a consistent format. You may need to pivot data or de-pivot it, or aggregate data. Self-service data preparation provides all these facilities.

However, the products in this category do not simply provide these facilities with a friendly user interface, they will also help with relevant tasks. For example, if you want to combine two sets of data the software will be able to suggest what fields you should join the data on, and it may recognise that you have missing demographic data for some of your customers and suggest where you could get that data from. Typically such products have both semantics and machine learning capabilities built-in. The former helps the software to make recommendations to you while the latter means that these recommendations improve over time. Machine learning can also make use of its knowledge as to how your colleagues are using the platform.

An additional capability is that data preparation platforms audit what you are doing. Not only does this have compliance implications (IT can see that you are not doing anything you shouldn’t be doing) it also means that if you decide to put a particular data preparation process into formal production (that is, build a formal IT process) then all the logic of the process has already been captured, making IT development efforts much simpler. Finally, the fact that IT can see what data is being analysed and how, means that IT can get a better handle on the sorts of services it might need to provide to the business.

This page has been archived and merged, please visit the Data Discovery and Catalogues or Trust in Data pages for new content.

Self-service data preparation is primarily aimed at business analysts or data scientists in the business units that are going to use it, because it allows them to prepare and perform ad hoc analyses without being reliant on IT. However, there are also major benefits for IT, not just in being able to monitor what users are doing but also in reducing IT workload. Without platforms of this type, IT will have to use integration tools (whether data integration or data virtualisation, data quality, data governance and other such products, all of which takes time and effort.

It is perhaps worth commenting that self-service data preparation platforms are, in effect, data governance platforms: it is just that the governance aspects are hidden from the end users.

This page has been archived and merged, please visit the Data Discovery and Catalogues or Trust in Data pages for new content.

This whole market is emerging. At present different vendors are targeting different sectors of the market: in particular business analysts versus data scientists, and data preparation versus data unification (where the emphasis is on merging very many datasets). There are certainly constituencies for all of these but over time we expect the distinctions between products to blur as different vendors extend their current capabilities into other areas.

This page has been archived and merged, please visit the Data Discovery and Catalogues or Trust in Data pages for new content.

There are a few major vendors, notably IBM, Informatica and Progress, that are active in this space while most of the others are smaller and/or start-ups. In this last category we would include Paxata, Trifacta and Tamr. There are also two vendors: Alteryx and Clearstory that go a step further than data preparation and actually some level of analytics as well. However, even these vendors partner with Tableau, Qlik and other business intelligence vendors for interactive functionality and extended visualisation.