Kx Systems
Last Updated:
Analyst Coverage: Philip Howard and Daniel Howard
KX was founded in Palo Alto, California in 1993. It is a business unit of FD Technologies (formerly First Derivatives), which acquired a majority shareholding in KX in October 2014. FD employs over 2,000 people worldwide and has operations in cities around the world, including London, New York, Stockholm, Singapore, Hong Kong, Tokyo, Sydney, Toronto, Philadelphia, Dublin, Belfast, Zurich, Seoul and San Francisco.
Similarly, KX has found traction across a wide variety of industries, with a particularly strong presence in financial services. In capital markets, where the company originally focused, it uses both direct marketing and channels, and it has a number of partners (situated worldwide) that provide sales, consulting, and OEM services.
It initially achieved success within the smart energy and utility sectors but has since added customers for sensor data in manufacturing, Formula 1 racing, telecommunications, cybersecurity and the aerospace sector. Going forward, the company is focusing on the development of vertical applications within these sectors.
kdb+
Last Updated: 3rd March 2016
Kx technology consists of the kdb+ database, the q language, a CEP (complex event processing) engine associated with the database, and various tooling products designed for non-specialist users. Kdb+ is a column-based, hybrid in-memory database with stream processing capabilities, primarily designed for analytic workloads. In so far as in-memory capability is concerned, we refer to it as "hybrid" because it uses in-memory processing as much as it can but recognises that in some cases it may be impracticable to load all relevant (typically historic) data into memory and that you therefore need to employ techniques that will not only leverage memory-based processing but optimise performance when not all the data can fit into memory.
The product's stream processing capabilities (which means that you can analyse very large quantities of information in-flight, in real-time) arise from the fact that kdb+ is tightly integrated with the product's development language q. This is a vector (array) processing language that is much more efficient than SQL and which can be used to develop analytic applications as well as for query purposes.
The company's historical approach to marketing has been both through direct marketing and via channels. It has a number of partners around the world that provide product sales, training and installation as well as first line support. Many of these partners have extended Kx's capabilities by offering specialised financial capabilities with things like graphical user interfaces for business intelligence and extended complex event processing. In particular, First Derivatives has built a suite of business intelligence products, including a visualisation tool, on top of kdb+.
In addition to these partners the company also has OEM partners. For example, 1010Data, a cloud-based data warehousing solution for financial service companies, is based on Kx technology. Datawatch is another Kx OEM client.
However, while this has been Kx's approach to financial markets it has been directly addressing big data opportunities outside the financial sector. As this becomes more mature we expect the company to follow a similar channel model.
Kx Systems is active in encouraging the use of its technology in the academic world and a number of universities in North America, Germany and the UK are using it.
Kx has an impressive customer list. In the financial sector users include Goldman Sachs, Morgan Stanley, Merrill Lynch, JP Morgan, Deutsche Bank and many others. Nine of the world's top ten financial institutions are clients and more than a hundred financial institutions and hedge funds leverage Kx's technology.
The company's move into the big data space and the Internet of Things is in its early days but it has already attracted a number of customers, include Purdue Pharma and Total Gas and Power UK.
Kdb+ stores data in a columnar database, which may be compressed, if required. The software implements a hybrid in-memory/disk architecture that is optimised to use whatever resources are available. Data [in] particular, stream processing often requires that historic data is needed for contextual purposes and, given that a year's of worth of tick data (for example) comprises something like 6 or 7TB then you really do need hybrid capability. As far as availability is concerned kdb+ stores memory images on disk to facilitate rapid recovery.
A further feature of kdb+ that is worth mentioning is the fact that it supports user-defined datatypes as well as built-in datatypes. In the latter case, time series support (which can be accurate down to nanoseconds) is implemented by means of specific datatypes for elements such as dates. Similarly, a relational table is a base datatype.
On the language side q is a declarative language with similarities to SQL. For example, it has functions such as SELECT statements and WHERE clauses as well as updates, deletes and so forth. Where it goes beyond other environments is in its mathematical functions, such as variances, and in the fact that developed applications are tightly integrated with the data and database. More technically q is a vector programming language (that is, it addresses vectors [arrays] rather than tables) that uses memory mapped files for numeric processing. This has important implications, not only in its own right (because you get better performance) but also because Intel is increasingly adding vector processing capabilities to its processors. As Kx Systems is a partner of Intel, the former aims to leverage each new piece of relevant functionality that Intel introduces, so that performance will continue to improve.
A further consequence of the fact that most programming is written in q rather than SQL is that there is not a native fit with third party BI and visualisation tools. However, First Derivatives have built these capabilities on top of kdb+ and there is also an ODBC 3 driver that allows kdb+ to be used in conjunction with Tableau and Excel.
A number of IDEs (integrated development environments) can be used in conjunction with kdb+. Because of the conciseness of the code, programs tend to consist of relatively few lines, which makes debugging relatively simple. Code is interpreted but the interpreter takes up not much more than 100K and this makes it easy for the environment to support many simultaneous processes. This small footprint not only means that installation is very fast, it also reduces the risks and costs associated with upgrades and maintenance.
Kdb+ runs on Solaris, Windows, Linux and OSX platforms and is currently in version 3.3 with version 3.4 to be available shortly. There is also a free-to-download 32-bit version that runs on practically everything.
In addition to the usual sorts of services one might expect (training, support and so forth), which are provided by both Kx Systems and its partners, the company runs a web community (www.kxcommunity.com). This provides details of events, jobs and meet-up groups as well as a discussion board, conversations and blogs. It also details free tools that are made available within the Kx community (whether by Kx or by a member of the community) and which you can download.
KX
Last Updated: 14th December 2021
Fig 1 - KX conceptual architecture
At its core, KX (or, more properly, KX Streaming Analytics) is a highly performant streaming analytics platform layered on top of a time-series database. It allows you to analyse streaming, historical, in-memory, and time-series data rapidly and simultaneously. In turn, this can enable faster and better decision-making by providing you with up-to-date, comprehensive, and intelligent information at your fingertips via dashboards, apps, alerts, and so on (conceptually illustrated in Figure 1). The platform also offers real-time stream processing and an in-memory compute engine, as well as various industry-specific solutions. The platform’s design is such that it offers a broad base out of the box, which can then be built upon using these solutions to tailor it to your particular needs.
The combination of stream processing, in-memory processing and persistent data storage means that KX provides what is effectively a Kappa architecture. Where it differs from many other potential providers of Kappa, or indeed Lambda, architectures – particularly where these are to be built on a variety of open-source offerings – is that KX offers a consistent, unified approach to supporting the combination of real-time and batch analytics that Lambda and Kappa architectures are aimed at, with a single code base.
Customer Quotes
“Through its ability to rapidly process vast amounts of time-series data, provide analytics in real-time, and integrate with oura machine learning pipelines, Kx is ideally placed to power our neuroscience platform.”
BrainWaveBank
“There are many valuable applications of satellite imagery across a range of industries, many of which are time sensitive and require powerful analytic processing. By combining our data and Kx technology we expect to be able to provide new and existing customers with unique and valuable insights.”
Airbus Defence and Space
KX is built on the Kdb+ database. Although KX refers to it (not without cause) as a time-series database, its under-the-hood architecture is columnar. On top of Kdb+, the company provides a built-in array processing language called q. The array-based nature of this processing means that it can work across both columns and rows simultaneously, making it much more efficient that traditional approaches. It is also worth noting the terseness of q, which is actually SQL-like. qSQL also exists as an explicitly SQL-like table querying syntax. Recent releases have also provided some support for SQL itself. In addition, a large number of connectors are available – more than a hundred in total – and you can call functions, methods and libraries written in Python, R, C and C++ from within q code. Integration with C# and Java is also provided, as are ODBC and JDBC drivers.
The combination of Kdb+ and array processing facilitates concurrent execution and parallelism, which feeds into the platform’s notably excellent benchmark performance. It’s suitable for large scale (100+ GB of data per day) deployments, and it has an impressively small footprint of less than a megabyte. This gives it significant scalability: it is equally capable of deploying to a centralised enterprise system as to the edge. Furthermore, it supports deployment across multiple machines with a distributed capability for scale-out in clustered environments. Resilience and automatic recovery are also provided, together with load balancing and replication.
KX has invested significantly into the cloud via KX Insights, which is essentially the platform’s cloud-first deployment option. Accordingly, it can be deployed across public and private clouds, including multi-cloud and hybrid environments, and it provides OpenShift-certified support for Docker and Kubernetes. It is available from the AWS, Google Cloud Platform, and Microsoft Azure cloud marketplaces as well as from KX directly.
Fig 2 - KX dashboards and visualisations
The company has also made a point of investing in machine learning (ML) and AI. To wit, KX provides an out-of-the-box ‘ML Toolkit’ that offers data pre-processing, model training, model scoring, and so on. Clustering, natural language processing, and support for external machine learning models are also available. Perhaps most notably, KX offers an ‘AutoML’ feature, which promises to help automate your machine learning processes through the use of customisable, end-to-end workflows. The platform also provides data preparation capabilities, native statistical libraries, and integration with various machine learning environments such as TensorFlow, Theano and others.
Other notable capabilities offered by the KX platform include real-time monitoring, alerting and network analysis. It also features dashboarding functionality and real-time visualisations, as shown in Figure 2.
There are a number of reasons to like KX. It boasts significant (and impressive) technical advantages, particularly in terms of performance and footprint. Although it was originally designed primarily to serve the financial services sector, these advantages apply just as well to (and the platform is compatible with) many different industries and environments, most notably IoT and the edge.
Moreover, KX can analyse multiple different kinds of data (most notably real-time and batch data) within a single platform, simultaneously; it leverages time-series capability as a core functionality (a rarity in the streaming space); it provides highly contextualised and intelligent insights on a continuous basis; and it offers substantial support for machine learning. This all results in the rapid delivery of analytics that are also robustly functional, comprehensive, and useful.
The Bottom Line
KX differentiates itself most notably with its high-end performance, miniscule footprint, and first-order time-series support. Accordingly, if you want to deploy a streaming solution to the edge, analyse streaming data (particularly sensor data) where time is a factor, or generate comprehensive analyses at speed across many different kinds of data, KX is more than worth a look.
Kx Systems kdb+ and Kx Technology
Last Updated: 28th February 2020
Mutable Award: Gold 2019
Kx is built on the kdb+ database, an in-memory columnar database with both streaming and time-series capabilities. You can optimise performance by combining the processing of in-memory data and data persisted to disk. Applications on kdb+ are built using q, a declarative, SQL-like language, with a strong mathematical library.
Q is a vector array processing language that supports the rapid, low latency analysis of time-series data. Vector array processing is also highly amenable to parallelism and concurrent execution. As a result, kdb+ offers parallel deployment across multiple partitions with a distributed capability for scale-out in a clustered environment. Resilience and automatic recovery are provided, with load balancing and replication both supported.
Fig 01 Typical Kx Solution architecture
In practice, the combination of stream processing, together with in-memory processing and persistent data storage, means that Kx provides a so-called Kappa architecture (a simplified version of a Lambda architectures), as shown in Figure 1, even though this is hidden under the covers. Where it differs from many other potential providers of Kappa, or indeed Lambda, architectures – particularly where these are to be built on a variety of open source offerings – is that Kx offers a consistent, unified approach to supporting the combination of real-time and batch analytics that Lambda and Kappa architectures are aimed at, with a single code base.
Customer Quotes
“Through its ability to rapidly process vast amounts of time-series data, provide analytics in real-time, and integrate with our machine learning pipelines, Kx is ideally placed to power our neuroscience platform.”
BrainWaveBank
“There are many valuable applications of satellite imagery across a range of industries, many of which are time sensitive and require powerful analytic processing. By combining our data and Kx technology we expect to be able to provide new and existing customers with unique and valuable insights.”
Airbus Defence and Space
As can be seen in Figure 1, Kx is not limited to providing an engine but also offers Control for Kx, Monitor for Kx, Analyst for Kx, Dashboard for Kx and various industry-specific solutions that reflect the company’s target markets. Some further relevant points include the fact that Kx has a small deployment footprint (600KB) so is suitable for embedding in edge devices; is available in the cloud as well as on premises; and includes an ODBC driver for connection to popular third-party dashboard platforms such as Tableau, as an alternative to the tools Kx offers.
Fig 02 Typical Kx for Monitoring architecture
Of the tools available, Analyst for Kx provides data preparation capabilities, native statistical libraries and integration with machine learning environments such as TensorFlow, Theano and others. You can also call python libraries from within q code, and you can use python to import third party machine learning models. Kx for Monitoring, on other hand, is specifically targeted at network, IT and similar environments where you need to monitor the state of your infrastructure. This is illustrated in Figure 2. It supports the real-time analysis of the network to display critical information and raise alerts, and it can combine this with historic information to support such things as trend analysis. Finally, there is also Kx Developer, which provides an IDE for q developers.
Kx has proved itself in what is unarguably the most demanding big data market: financial trading and risk management. The company has been targeting IoT and related environments over the last few years and has had some notable success across a range of applications, where the collection and analysis of time-series based data is important. Compared to other products in the time-series space Kx not only offers superior performance but also support for machine learning: something that is sadly lacking from most of its competitors.
One area where the company’s capabilities are thin, is in support for geo-spatial capabilities, which are often combined with time-series in IoT environments. The product does have the ability to store latitude and longitude co-ordinates and it does have a client overlaying images onto a mapping interface but there is no support for vector maps or spatial indexes. To be fair, this is not a strong point for many of the transactional time-series databases, but we would like to see the introduction of further capabilities in this area.
The Bottom Line
Kx is probably the leading provider of time-series data to the financial services (especially capital markets) sector. For an IoT perspective it offers a small deployment footprint, a single platform for streaming analytics and time-series capabilities, plus the optimisation of in-memory and persisted data processing. It therefore fulfills the requirements for embedding in edge and gateway devices as well as for more centralised functions.
Mutable Award: Gold 2019
Research
Streaming Analytics (2021)
KX (2021)
Time-Series and Temporal databases and analytics
