Event stream processing and Operational BI

One of the hot topics in the business intelligence world over the last year or so has been operational BI. Let me first make a stab at describing what is meant by this term. In effect, it differentiates between the analysis of what happened in the past, the analysis of what is happening now, and predictive analysis of what will happen in the future (which, incidentally, is based on what happened in the past), with operational BI sitting in the middle category.

Now, the key to operational BI is that it needs to be at least close to real-time. This will depend on circumstances: if ice cream sales are going through the roof at your local supermarket and they are running out, because the temperature just passed 90°, then you need to replenish stocks now and not wait until tomorrow. On the other hand, some things aren’t quite so urgent.

There are therefore a range of operational BI approaches that may be more or less suitable, depending on the environment. Colin White has written extensively on this subject for DM Review and I do not intend to go over the ground he has already covered. Briefly, the solutions he outlines include replication, federated query, an operational data store (or data warehouse appliance) and the use of change data capture. However, there is a further category, which is event stream processing (ESP) which, of all these technologies, should provide the most real-time of business intelligence capabilities.

Most ESP and CEP (complex event processing—a superset of ESP) vendors focus on applications such as algorithmic trading, RFID, fraud detection and so forth. However, there is one supplier (and possibly a second—Syndera—but I have not had a chance to evaluate its solution yet) whose specialisation is on business intelligence. This is SeeWhy.

The big difference between SeeWhy and conventional ESP/CEP vendors is that it has a big focus on metrics. Moreover, it is not just about detecting anomalies (which may be either threats or opportunities) it is also about fine granularity reporting. For example, suppose that two stores both sell 200 of a particular item over the course of a month: how do you know whether there was an even spread of sales? In one store sales may have been evenly distributed over the month, in the other the sales graph may have looked more like an alpine range—selling out completely each time that stock is replenished but then having to wait so many days for new stock to arrive and then promptly selling out again.

With conventional tools you simply cannot track against what may be tens of thousands of products over hundreds of stores. Using ESP technology you can. But SeeWhy (with the possible exception of Syndera) is the only vendor that I know of that has built the BI capabilities on top of its engine to enable this automatically. Indeed, it even has its own data mining capability (so SeeWhy supports predictive as well as operational BI) to analyse the information gathered in more depth if you need to, though you could also plug in SAS or SPSS if you preferred.

Going to operational BI, the sort of approach adopted by SeeWhy is what operational BI is all about. While it will not be for everybody, for those that genuinely need to be on top of their business on a minute-by-minute basis, this is the sort of technology you should be looking at.