An operational approach to managing Big Data

Some time ago, I expressed concern about whether current IT approaches could cope with the need of the emerging “universe of things” – machines and sensors in machines, all generating vast amounts of data all of the time.

Now, RTI, one of the companies I mentioned in that blog, has announced its new product suite, RTI Connext, a family of products designed to connect real-time applications across an enterprise. Connext Micro is for embedded systems in small devices; Connext DDS is for full-function applications using the OMG DDS data-centric architecture standard; Connext Messaging is for general purpose real-time applications and also supports JMS (Java Messaging Service); and Connext Integrator provides adapters to conventional SOA-based and discrete IT applications. The whole is held together with the RTI DataBus.

In essence, these products can marry real-time “operational systems” approaches, using data-driven and event-processing techniques and are used to manage jet engines in planes during flight and operational warships, to conventional business information processing systems.

This could become extremely important if, as I anticipate, we are approaching a bigger “inflection point” crisis in data volumes than some people are contemplating. There are lots of clever ways to make existing technology more efficient, in order to deliver near-real-time results from the anticipated growth of data in today’s IT systems. But what if, tomorrow, entirely new systems emerge and add to what already looks like exponential information growth?

We can just about cope with, say, RFID systems generating data for human-scale processing. But what if every machine out there produces data for interpretation by other machines (the Semantic Web is probably relevant here, but it seems to me that this is more concerned with meaning and ontologies than real-time performance); which, in turn, generate more data as a result of these interactions; which have, in turn, to be analysed so as to support strategic human-level decision-making? Which then feeds back data into the system that machines interpret as instructions, generating more data as they respond and change their state… “Data in motion” is starting to be as important as data in databases.

We will probably need new architectures for this – but we will probably still want to reuse human-scale technologies where appropriate. The neat thing about RTI Connext, I think, is that it supports hybrid solutions, so you can introduce new approaches as and when they are necessary, not in an anticipatory “rip and replace” effort which will probably destroy your business in the short term…

So, Connext is worth looking at, I think, together with other innovative approaches to handling really Big Data, such as Pervasive’s DataRush accelerator. I’ve seen a Siemens case study around real time management of wind farms which uses the RTI approach because conventional technologies simply can’t cope today; but, for most of us, we should be thinking of the implications of data-centric, event-driven, publish-and-subscribe approaches to handling Big Data on our future business automation strategies.