Whither analytics?

Demand for analytics is exploding. Not surprisingly, there is a surge of new products and technologies designed to support that interest. Some of these new products or technologies are at the platform level and others are more in the way of development tools while some have elements of both.

In the first category are in-database analytics supported by various data warehouse vendors, though some of the warehouse vendors also provide their own analytic functions in-database so that you don’t actually have to rely on third party data mining tools. Also in this category are databases like Hadoop.

In terms of development, data mining has got much more interesting in the last few years with newer companies like Fuzzy Logix making an entrance to compete with the likes of SAS and SPSS, as well as open source data mining vendors like KNIME. Fuzzy Logix in particular is interesting because of its success in forging partnerships with the likes of Sybase, Netezza, Microsoft and Aster Data.

Also on the development side we have seen a growth in interest in MapReduce and R, though in the former case there is still much discussion over whether MapReduce should be integrated with SQL or left separate; the argument for the latter being that SQL developers are not usually conversant with the languages that can be used to exploit MapReduce, and vice versa. In any case, support for these is now extending beyond the usual suspects so that Netezza, for example, will be supporting them both in its forthcoming i-Class release, while the likes of SAS also support the use of MapReduce.

Hadoop is also gaining traction and will continue to do so now that IBM has announced that it will be providing professional services support for it. Most recently, there has also been the announcement by Talend that it has integrated with Hadoop. Moreover, we are now starting to see the appearance of higher level products, like Datameer, which provides an analytic development environment that sits on top of Hadoop and hides its complexities from developers.

And then you have the third category, which includes the data warehousing vendors that are providing their own in-database analytics capabilities (Aster Data, Netezza, Sybase and so on) as well as products like the latest release of Pervasive DataRush (which uses KNIME). And, of course, vendors like SAS and Fuzzy Logix are embedding their capabilities within these data warehousing products, which significantly extend the functions provided by the warehousing suppliers.

Now, where is all this headed? The short answer is that a lot of these companies are looking for (and finding) ISVs and VARs to develop analytic applications. And most of these applications are being targeted at the mid- and small enterprise market, and at departmental solutions, in large part because SAS is so dominant in the enterprise analytics market. Of course, SAS has a significant footprint in the mid-market (which it categorises as companies with less that $500m in revenues) but it is a big market and there is lots of space for innovative and specialist suppliers to create niche markets, especially with vertically oriented products.

I therefore expect an explosion of new analytic applications coming onto the market, based on in-database analytics running on a plethora of data warehousing platforms. Just as data warehousing was boring a few years ago before Netezza and its followers came into the market, but has become interesting since, so I expect the market for analytic applications to blossom. SAS has been dominant in this space for a long time. I expect it to remain so but I also expect it to get a lot more competition.