Thoughtspot, a truly disruptive innovation that will change how you perceive BI

Business Intelligence has long been one of the great oxymorons of our time, we talk about its importance, and how it should be integral to everything as we now live in a knowledge based economy; but the reality is that the adoption rates are low, the overall levels of satisfaction are low, and the answers the industry comes up with fail to change that pattern. Then along comes Thoughtspot and turns everything on its head and offers a real breakthrough in what is possible and gives a glimpse into what will surely be the future.

The BI market is huge it turns over some $70 billion a year, but I know of very few who are happy with what they have and feel they have value for money from their investment. The reasons for that are numerous but I would highlight as the critical ones, the barriers to ease of use, the cost of implementation when compared to the value received, the focus within the industry of technology and not on how to adopt it to the business processes and the yawning gap between the hyperbole of the marketing speak and what people actually see on the ground. There have been so many cul de sacs of thinking, for instance over the last few years we have had self service being thrust at us, but I know of no sane business person who wants to understand data models, and BI technologies let alone get into SQL, we are now being told that the answer is a mixture of visualisation and data science, but in the next breath I get told there is a chronic shortage of data scientists and that good designers of visualisations are few and far between.

At heart no one actually wants to do it themselves, as a wise man told me yesterday its like DIY in the home, very few people want to go out and buy a drill, drill bits and wall plugs, what they want is a hole in the wall ready to take a screw, and it’s the same in the world of analytics, no one wants to be a BI expert they want to have answers that help them to make better decisions in their business domain. So does that mean people are just technophobic luddites and we should dumb down to pander to them? Certainly not, you have only to look at the success of Google to understand how readily people are to use technology to help them answer questions. So there has to be a better way.

In essence what ThoughtSpot have done is to create an AI driven analytics engine, which is exposed via a simple Search query. At the heart is a high performance database, which is extremely scalable (currently the largest database is millions of rows and a couple of tens of terabytes so non trivial) in which all of the data is indexed, the AI search engine is looking for trends, anomalies, leading indicators and segments in the data, and is learning the whole time from the sorts of queries that are asked. It learns to not only answer commonly asked questions but from a combination of learning what is asked for and the patterns in the data itself will identify things that should be asked. Although it is doing this analysis in the background it is not completely opaque the lineage of the queries is transparent and can be traced and manually replicated, which was always one of the problems with the early use of neural networks and genetic algorithms, where often the answers looked plausible but no one could work out where they came from!

None of this would work if it were not possible to gather data from all the sources found in an enterprise so there are connectors to all of the standard stores of corporate data from Sales Force to Hadoop to Oracle, SQL Server, IBM etc etc and the other databases.

This is not just a big brother technology where the answer is what the computer says, it is about an ongoing interaction between the human users and the machine learning. This is the first substantiation I have seen of what I envisaged cognitive analysis would look like and I shall be following how this developers very carefully.