The EDW vision

Traditionally, enterprise data warehouses (EDWs) were regarded as systems of record. Thought of simplistically, queries were either run directly off the EDW or from a data mart that referred back to the EDW as and when necessary (hence its status as a system of record). However, it is now clear that that definition of an EDW no longer holds good or, at least, is not what many people mean when they refer to an EDW.

An EDW is increasingly referred to as the place that will support what HP, for example, refers to as “ubiquitous BI”. What it means by this is the place that people will go to run all of their queries. However, that needs a little explaining.

Traditionally, the people that have mostly used data warehouses have been business analysts and similar who used the warehouse frequently, did sophisticated slicing and dicing, ran analytic queries that were more or less complex and did data mining. These people represent a relatively small percentage of the user population but they did a lot of work. What is being suggested in the ubiquitous BI world is that they will be joined by all the other corporate staff, all of whom will be running queries and/or reports against the warehouse, albeit that these will be infrequent, simple, short queries.

Now, EDWs have always had to cope with the fact that some queries are simpler and shorter than others and that’s why the various vendors offer facilities such as scheduling and prioritisation that support mixed query workloads. However, what is different about the ubiquitous BI scenario is the scale that is implied by allowing thousands or tens of thousands of users access to the warehouse.

The first point to note is that this vision of ubiquity is a stick to beat the data warehouse appliance vendors with. These companies: Netezza, DATAllegro, Kognitio and the like are all introducing facilities to support mixed query workloads if they have not done so already. However, none of these suppliers would suggest (at least today) that they can support the sort of user community that the traditional suppliers (plus HP) are contending that they can manage in the new world of EDWs.

So, the new EDWs play into the hands of the big boys. But, how much of this vision is reality and how much of it is hype: is it realistic or is it just a stick?

Unfortunately for the major data warehouse vendors the answer to this question does not lie in their own hands. The reality (or otherwise) of ubiquitous BI lies in the hands of the BI vendors themselves, and there is no clear evidence that they can deliver on this promise. After all, they have actually been promising this same thing for a decade or more (albeit under different headings, such as the democratisation of data) and the amount of BI shelfware in the world suggests that they have clearly failed to deliver.

As it happens, I believe that the BI market is going through a similar transition phase to that of the data warehouse and it may, repeat may, once it has come through this period, be able to offer such compelling functionality and capability that ubiquitous BI can become a reality. I would like to think so. However, it certainly isn’t the case today and it is by no means clear that it will be tomorrow.

In other words, the vendors pushing the new EDW concept are putting the cart before the horse. They are promulgating the idea that you must have one of their systems (at the expense of appliance-based solutions) on the basis that only they can offer support for ubiquitous BI even though companies are not, in general, implementing such a thing or feel that such a thing is actually available. In other words the new EDW is a vision: a nice vision to be sure, but a vision of what may, just may, happen in the future is not something I would want to bet my business on.