Splunk: are you imaginative enough to realise its potential?

I was first introduced to Splunk last year and, once I got used to its name, quickly came to realise that this was something that had amazing potential. Splunk is a leader in its market, which is the provision of real-time operational intelligence. What it does is collect, index and make available for exploitation all of the machine data that is being generated as part of the Big Data explosion that is going on all around us. What that means is that, as machines and processes increasingly have monitors attached at every significant node, the mass of data that tells us what is happening, or what is not happening and, having collected it, Splunk indexes it so that it is turned from a mass of data into information, which can then be exploited.

Splunk has just announced a new version, Enterprise 4.3, that adds to its already impressive capabilities. In keeping with all tools designed to operate in the real world of Big Data, Splunk already provides scaling, speed and cost effective operations to enable you to tackle all of that machine data in a way that is affordable and makes it compelling to tackle the problem of ensuring that you do deliver excellent service to the letter of an SLA, and it now does that even better. In 4.3 you have that other great essential of the current crop of BI tools, mobility. It has a non-Flash user interface, which delivers the power of Splunk to you wherever you may be, and, of course, being non-Flash means you can use your tablet of choice, even if that is an iPad. It is even more powerful, being able to handle up to 10x more concurrent users, and it’s faster, being able to run search up to 10x faster on the same platforms as the previous edition and, finally, it is more user friendly, with its dashboards now being editable by business users and executives.

Splunk is probably one of the best-kept secrets going around at present. It is used wherever there are masses of data, and service is a business-critical issue; so the technical community in ISPs, ecommerce vendors, energy companies, financial service companies etc. rely on it. But it opens the door to so much that is required to deliver really effective Big Data solutions. Big Data is about getting a true 360-degree view of the world, integrating that data, and acting on it, to create new business opportunities. The explosion of data that represents Big Data is not caused by more of the same, there are not more transactions in the world now than there were a few years ago; in fact, with the current economic climate there are probably less, but there are now more data sources. There are, obviously, the standard transactions that form the basis of most traditional data warehouses, and to that we can now add analysis from the likes of social media, such as Facebook and Twitter, to understand more about how people interact, and view things as a social group; and we have machine data, which, in many ways, underpins everything else. There is not much point in understanding what we buy and sell, or what we think of things, if we do not understand the process that is the basis of that transaction or social commentary. This is where Splunk sits, giving that insight.

With the new features of Splunk, its mobility, its adaptability without needing to edit XML, its ability to access real-time and historical data in a single step, its speed and manageability, it is no longer a tool that should just be thought of as a back-office essential. This is information that is vital to ensuring that profitable customers are provided with excellence of service and retained; for providing the platform to extend the reach of your product range to other potentially profitable and loyal groups. So Splunk should be a true enterprise tool, seen as part of the corporate arsenal, providing competitive advantage, used as much by Marketing and Customer Service as by engineers.

The advantage that Splunk offers is that you can start small, build confidence and grow. It is not about giant leaps of faith. Because of its cost model you scale up as the business benefits are realised. We are not talking about the complexity of a full blown real time Complex Event Processing solution, which is a giant leap of faith; this is a logical set of building blocks that will allow you to get there incrementally and, with Splunk, you will be able to go all the way up to being able to leverage the data on Hadoop clusters in the near future, so this is not a path with limits.

The big question is, can those outside of the traditional base of Splunk realise this potential and be in the vanguard? Or will Marketers and others in the enterprise wait to be told by IT that this is technology you should look at, by which time a competitor with more imagination may have stolen a march on you. These are exciting times, and I believe that Splunk is part of the answer to our finding ways of reinvigorating our economic activity through innovation and effective customer focus.