Edge appliances and the evolution of data warehousing

Traditionally, there have been two types of system that made up the data warehousing landscape: enterprise data warehouses (EDW) and data marts. However, this position is changing.

First of all, the traditional EDW is evolving. Historically, it was a system of record. Its other characteristics: batch loading, numeric data only, limited number of users and so on, were all secondary to its being a system of record. However, that position is being usurped by master data management systems. This is not to say that this role will disappear soon from the EDW but it certainly will over a period of time, at least for most organisations.

What will take its place? Well, the new EDW (referred to by some EDW 2.0—yeuch!) will not be a system of record per se but it will support very larger numbers of users, it will support real-time operations and it will support unstructured data (but see below) as well as numeric data.

Data marts will not go away. They will continue to be used for departmental and other requirements. In addition, I think that the old style EDW, minus the system of record bit, may effectively be used, at least in some companies, as a sort of specialised data mart for such things as data mining across departments.

However, where we will see the biggest change is in the introduction of so-called Edge Appliances. What do I mean by Edge Appliances? In effect, these are appliances that have had application software designed specifically to exploit the appliance architecture and which are then pre-loaded into the appliance along with the operating system and database. A good example is Kognitio’s Pricing Appliance, which has been designed (in conjunction with RateIntegration’s PriceMaker Enterprise Pricing Server) to provide revenue assurance, tariff planning, and margin and cost analysis for the telecommunications sector. Another example is LogLogic’s Log Data Warehouse appliance, which is an appliance that has been specifically built for analysing (network, database, whatever) log data. We can expect to see more Edge Appliances introduced, from various vendors, in the not too distant future.

One can easily imagine many other such specialised appliances built (unlike LogLogic) on top of generic data warehouse appliances: for example, appliances for analysing RFID data, healthcare information, clickstream and GPS data or, more prosaically, particular analytical applications or sets of such applications. At the moment, as I mentioned above, it looks as if the trend is towards including textual data within the data warehouse, especially when you want to combine text and numeric data. However, it is perfectly possible that we will see the emergence of Edge Appliances that do this sort of thing: for example, applications in the insurance industry comparing claim form data with statistical data.

The emergence of Edge Appliances will also see a shift in the market, because most of these products will be joint ventures with one company, as in the case of the Kognitio Pricing Appliance, providing the warehouse appliance functionality and the other providing the relevant application software. In effect, this will mean the appliance vendors building an indirect channel. To date, most of these vendors (Netezza, DATAllegro, Kognitio, Vertica et al) have very little experience of building and managing an indirect channel as their sales models all revolve around direct sales. So this represents a new challenge that could change market dynamics. In addition, Hewlett-Packard does of course have such experience. While its pricing model for NeoView is not likely to prove attractive to the channel right now, this could change if H-P sees potential in Edge Appliances, and if it can compete on performance terms.

To conclude: I believe that the market is evolving from a two-tier EDW/data mart model to a three tier EDW/data mart/Edge Appliance model. My prediction for 2008 (Christmas comes earlier every year!) is that we will hear much more about Edge Appliances.