Merchant relational databases: over-engineered and out-of-date?

Michael Stonebraker et al have just published a paper on the problems facing standard relational databases and on the potential of a new database development that is being pursued at MIT, called H-Store.

The argument goes as follows. First, we know that standard relational databases, which aim to provide all things to all men, do not provide as good performance (by a considerable margin) as specialised data warehousing products. Secondly, we know that they are similarly unsuitable for high-performance stream processing. And, thirdly, we know that they are not best-suited to text processing. We also know, though this is not mentioned by Stonebraker, that there are better ways of handling fixed data (that is, data that cannot be changed) such as bids on eBay, where products such as Oracle’s BerkeleyDB offer superior performance at lower cost. And there are a number of other environments where traditional approaches are sub-optimal.

In other words the only space where the traditional relational database vendors can claim to have any advantage over their competitors is in OLTP. And even that might be disputed by ANTs, whose alternative approach to locking provides superior performance and, indeed, by the likes of Software AG (Adabas). Leaving them aside, however, Stonebraker’s argument is that the merchant databases were originally designed, at least in part, to cater for the constraints placed upon them by the hardware and operating systems that were prevalent when their architectures were first put in place 20 or more years ago. And, moreover, those architectures have not been substantially or significantly revised since that time. As a result, they use obsolete techniques and design factors that are no longer relevant and that these databases really need a complete re-write.

The question therefore, is whether a relational database based on modern design principles and taking into account the facilities available from modern hardware and operating systems can out-compete the traditional database platforms for OLTP just as they can in every other area? This is what Stonebraker and his colleagues have set out to discover/prove with the development of H-Store; and the initial results suggest that the supposition is correct. In partial (H-Store is not yet complete) TPC-C tests, H-Store was able to outperform one of the commercial databases by 82 times. Moreover, the hardware it was running on was similar orders of magnitude lower in price.

So it looks as if Stonebraker is right and, to be honest, I’m not surprised: the arguments outlined above are logical and make sense. It is also worth commenting that Stonebraker does not think that SQL is necessarily the best approach for writing stored procedures in a specialised OLTP environment where, by definition, you don’t have ad hoc queries. To-date his team has been using C++ for this purpose but they plan to move to Ruby on Rails.

What can we conclude from all of this? Firstly, that there is no specialised function for which the merchant databases excel. They are a bit like middle order batsmen (apologies to those of you who don’t understand cricket) who bat a bit, bowl a bit and field quite well: all-rounders who are not exceptional at anything. Or, to make a different analogy: like a family saloon that has space for the kids and the dog and offers reasonable comfort at reasonable speed with reasonable mpg—but it ain’t a Rolls Royce or, for that matter, a Ferrari or an SUV.

Secondly, it is easy enough to predict that Stonebraker will shortly be setting up a new company to market whatever commercial name they come up with for H-Store. I will await that introduction with interest. My only piece of advice would be that it would make sense to support SQL with H-Store even if Ruby on Rails is offered as a preferred option, simply to make H-Store easier to take to market. I realise that this is crass commercialism but there you go.