Kx Systems kdb+ and Kx Technology

Kx is built on the kdb+ database, an in-memory columnar database with both streaming and time-series capabilities. You can optimise performance by combining the processing of in-memory data and data persisted to disk. Applications on kdb+ are built using q, a declarative, SQL-like language, with a strong mathematical library.

Q is a vector array processing language that supports the rapid, low latency analysis of time-series data. Vector array processing is also highly amenable to parallelism and concurrent execution. As a result, kdb+ offers parallel deployment across multiple partitions with a distributed capability for scale-out in a clustered environment. Resilience and automatic recovery are provided, with load balancing and replication both supported.

Fig 01 Typical Kx Solution architecture

In practice, the combination of stream processing, together with in-memory processing and persistent data storage, means that Kx provides a so-called Kappa architecture (a simplified version of a Lambda architectures), as shown in Figure 1, even though this is hidden under the covers. Where it differs from many other potential providers of Kappa, or indeed Lambda, architectures – particularly where these are to be built on a variety of open source offerings – is that Kx offers a consistent, unified approach to supporting the combination of real-time and batch analytics that Lambda and Kappa architectures are aimed at, with a single code base.

Customer Quotes

“Through its ability to rapidly process vast amounts of time-series data, provide analytics in real-time, and integrate with our machine learning pipelines, Kx is ideally placed to power our neuroscience platform.”
BrainWaveBank

“There are many valuable applications of satellite imagery across a range of industries, many of which are time sensitive and require powerful analytic processing. By combining our data and Kx technology we expect to be able to provide new and existing customers with unique and valuable insights.”
Airbus Defence and Space

As can be seen in Figure 1, Kx is not limited to providing an engine but also offers Control for Kx, Monitor for Kx, Analyst for Kx, Dashboard for Kx and various industry-specific solutions that reflect the company’s target markets. Some further relevant points include the fact that Kx has a small deployment footprint (600KB) so is suitable for embedding in edge devices; is available in the cloud as well as on premises; and includes an ODBC driver for connection to popular third-party dashboard platforms such as Tableau, as an alternative to the tools Kx offers.

Fig 02 Typical Kx for Monitoring architecture

Of the tools available, Analyst for Kx provides data preparation capabilities, native statistical libraries and integration with machine learning environments such as TensorFlow, Theano and others. You can also call python libraries from within q code, and you can use python to import third party machine learning models. Kx for Monitoring, on other hand, is specifically targeted at network, IT and similar environments where you need to monitor the state of your infrastructure. This is illustrated in Figure 2. It supports the real-time analysis of the network to display critical information and raise alerts, and it can combine this with historic information to support such things as trend analysis. Finally, there is also Kx Developer, which provides an IDE for q developers.

Kx has proved itself in what is unarguably the most demanding big data market: financial trading and risk management. The company has been targeting IoT and related environments over the last few years and has had some notable success across a range of applications, where the collection and analysis of time-series based data is important. Compared to other products in the time-series space Kx not only offers superior performance but also support for machine learning: something that is sadly lacking from most of its competitors.

One area where the company’s capabilities are thin, is in support for geo-spatial capabilities, which are often combined with time-series in IoT environments. The product does have the ability to store latitude and longitude co-ordinates and it does have a client overlaying images onto a mapping interface but there is no support for vector maps or spatial indexes. To be fair, this is not a strong point for many of the transactional time-series databases, but we would like to see the introduction of further capabilities in this area.

The Bottom Line

Kx is probably the leading provider of time-series data to the financial services (especially capital markets) sector. For an IoT perspective it offers a small deployment footprint, a single platform for streaming analytics and time-series capabilities, plus the optimisation of in-memory and persisted data processing. It therefore fulfills the requirements for embedding in edge and gateway devices as well as for more centralised functions.