kdb+

Kx technology consists of the kdb+ database, the q language, a CEP (complex event processing) engine associated with the database, and various tooling products designed for non-specialist users. Kdb+ is a column-based, hybrid in-memory database with stream processing capabilities, primarily designed for analytic workloads. In so far as in-memory capability is concerned, we refer to it as “hybrid” because it uses in-memory processing as much as it can but recognises that in some cases it may be impracticable to load all relevant (typically historic) data into memory and that you therefore need to employ techniques that will not only leverage memory-based processing but optimise performance when not all the data can fit into memory.

The product’s stream processing capabilities (which means that you can analyse very large quantities of information in-flight, in real-time) arise from the fact that kdb+ is tightly integrated with the product’s development language q. This is a vector (array) processing language that is much more efficient than SQL and which can be used to develop analytic applications as well as for query purposes.

The company’s historical approach to marketing has been both through direct marketing and via channels. It has a number of partners around the world that provide product sales, training and installation as well as first line support. Many of these partners have extended Kx’s capabilities by offering specialised financial capabilities with things like graphical user interfaces for business intelligence and extended complex event processing. In particular, First Derivatives has built a suite of business intelligence products, including a visualisation tool, on top of kdb+.

In addition to these partners the company also has OEM partners. For example, 1010Data, a cloud-based data warehousing solution for financial service companies, is based on Kx technology. Datawatch is another Kx OEM client.

However, while this has been Kx’s approach to financial markets it has been directly addressing big data opportunities outside the financial sector. As this becomes more mature we expect the company to follow a similar channel model.

Kx Systems is active in encouraging the use of its technology in the academic world and a number of universities in North America, Germany and the UK are using it.

Kx has an impressive customer list. In the financial sector users include Goldman Sachs, Morgan Stanley, Merrill Lynch, JP Morgan, Deutsche Bank and many others. Nine of the world’s top ten financial institutions are clients and more than a hundred financial institutions and hedge funds leverage Kx’s technology.

The company’s move into the big data space and the Internet of Things is in its early days but it has already attracted a number of customers, include Purdue Pharma and Total Gas and Power UK.

Kdb+ stores data in a columnar database, which may be compressed, if required. The software implements a hybrid in-memory/disk architecture that is optimised to use whatever resources are available. Data [in] particular, stream processing often requires that historic data is needed for contextual purposes and, given that a year’s of worth of tick data (for example) comprises something like 6 or 7TB then you really do need hybrid capability. As far as availability is concerned kdb+ stores memory images on disk to facilitate rapid recovery.

A further feature of kdb+ that is worth mentioning is the fact that it supports user-defined datatypes as well as built-in datatypes. In the latter case, time series support (which can be accurate down to nanoseconds) is implemented by means of specific datatypes for elements such as dates. Similarly, a relational table is a base datatype.

On the language side q is a declarative language with similarities to SQL. For example, it has functions such as SELECT statements and WHERE clauses as well as updates, deletes and so forth. Where it goes beyond other environments is in its mathematical functions, such as variances, and in the fact that developed applications are tightly integrated with the data and database. More technically q is a vector programming language (that is, it addresses vectors [arrays] rather than tables) that uses memory mapped files for numeric processing. This has important implications, not only in its own right (because you get better performance) but also because Intel is increasingly adding vector processing capabilities to its processors. As Kx Systems is a partner of Intel, the former aims to leverage each new piece of relevant functionality that Intel introduces, so that performance will continue to improve.

A further consequence of the fact that most programming is written in q rather than SQL is that there is not a native fit with third party BI and visualisation tools. However, First Derivatives have built these capabilities on top of kdb+ and there is also an ODBC 3 driver that allows kdb+ to be used in conjunction with Tableau and Excel.

A number of IDEs (integrated development environments) can be used in conjunction with kdb+. Because of the conciseness of the code, programs tend to consist of relatively few lines, which makes debugging relatively simple. Code is interpreted but the interpreter takes up not much more than 100K and this makes it easy for the environment to support many simultaneous processes. This small footprint not only means that installation is very fast, it also reduces the risks and costs associated with upgrades and maintenance.

Kdb+ runs on Solaris, Windows, Linux and OSX platforms and is currently in version 3.3 with version 3.4 to be available shortly. There is also a free-to-download 32-bit version that runs on practically everything.

In addition to the usual sorts of services one might expect (training, support and so forth), which are provided by both Kx Systems and its partners, the company runs a web community (www.kxcommunity.com). This provides details of events, jobs and meet-up groups as well as a discussion board, conversations and blogs. It also details free tools that are made available within the Kx community (whether by Kx or by a member of the community) and which you can download.