Events and other things

The advent of IBM’s System S (which I shall discuss in more detail in a subsequent article) raises interesting questions about the nature of event and stream-based processing. The reason for this is that the early adopters of System S are doing some interesting things with the product. For example, IBM has initial users processing sonar data, doing real-time facial recognition against video and CCTV footage, processing the high frequency radio spectrum and so on.

This is not to say that System S is not used in more traditional areas. For the more usual algorithmic trading, for example. Perhaps the most heart-warming is a hospital implementation where it is being used in intensive care to monitor premature babies. Apparently (I claim no expertise in this area) premature babies are at significant risk of blood poisoning and traditionally the first warnings of this are when it is recognised by experienced nursing staff. However, using appropriate algorithms, System S has been shown to recognise these symptoms 6 to 24 hours earlier. This not only helps to save lives but actually results in cost savings because the babies can be discharged from hospital earlier than would otherwise be the case.

Now, traditionally in the CEP (complex event processing) market we have thought about both discrete events and streams of data. But these can effectively be treated in the same way since you can regard a stream of data as a series of events with only a very short time lag between each event. This is one reason why the arguments for and against using SQL-based languages as opposed to purpose-built event processing languages has not been resolved, precisely because if you treat a stream of data as made up of individual events then it is perfectly feasible to treat the latter using a set-based approach such as SQL.

However, System S is a game changer not because it is processing streams of data but because things like video, sonar and microwaves are unstructured as opposed to structured and these are not requirements that have been generally addressed by typical CEP vendors. To be sure there are CEP solutions that use video data for automated number plate recognition, for example, but these are structured. I suggest that SQL won’t hack it when it comes to processing streams of unstructured data which is, of course, why System S is not SQL-based.

However, before I pronounce further we should consider that System S is by no means the first product to be able to process video data for facial recognition, for example. In this case, there are data warehousing platforms that can perform that sort of function. However, what they cannot do, is to process these in real-time.

So let me tell it as it is: System S is (as far as I know) unique in being able to process real-time structured and unstructured event-based and streaming data. As a result IBM will carve out a significant portion of this emerging market for itself.