Vertica at TDWI

Vertica, along with various other companies, has made a major announcement at this week’s TDWI, namely with respect to Vertica 4.0. This is a major release by any measure with the product being in beta today and general availability scheduled for the second quarter.

There are a number of major themes in this release, of which perhaps the most important is the workload management capability that is being introduced. This works in a similar manner to Teradata, in the sense that it is based on resource pools that are assigned to either users or types of tasks. The aim is to support any workload with any mix of query types and it will enable Vertica to go after enterprise data warehouse requirements where it could not previously compete.

A second major feature is support for Unicode. Historically, the product was limited to English only and this, again, was a limiting factor, which has now been removed.

Thirdly, Vertica has tackled one of the issues that the product formerly had, namely memory overflows. In order to make writing to disk more efficient, Vertica has what might be thought of as in-memory cache (actually a Write Optimised Store) into which data is loaded prior to it being committed to disk (the Read Optimised Store). However, a problem was that the memory could become full and cause queries to queue (or get rejected) before the software had got around to writing to disk. In this release this process is automated. You can still set your own parameters but if a potential overflow is detected then the software will override those settings to initiate writing to disk. In an associated development the company has also introduced a more efficient mechanism for processing deletes and updates in the storage layer.

Fourth, Vertica has extended its SQL support to add new capabilities. For example, there is now a facility for calculating moving averages, something which is important in capital markets but is difficult to do in standard SQL (though it is relatively easy in MapReduce). There are also facilities to support time series, including gap filling (when no trade occurs within a time interval) and web sessionisation. There are also more general improvements to query capabilities. For instance, Vertica could not previously perform a full outer joinyou had to perform a left join and then a right join and combine the resultsbut now it can. Associated with all of this are improvements to the optimiser, not only to recognise all of these additional capabilities but also for its own sake. Thus the optimiser can perform re-segmentation on the fly, for example.

As I said at the outset this is a big release. It adds significant new capabilities for both vertical markets and international markets, while also extending its breadth of capabilities. Given that the company’s acquisition rate is currently around 15 to 20 new customers per quarter we would expect this to further ramp up.