observIQ: Optimising the observability data pipeline - An open source and open telemetry approach

observIQ is an interesting player in the observability space. On its website, observIQ says that it builds “fast, powerful, and intuitive next generation observability technologies for DevOps and ITOps – built by engineers for engineers.”

It is a firm believer in Open Souce (which we like) and supports OpenTelemetry as do an increasing number of companies in the Observability space (such as Dynatrace and Splunk, for example). Open Telemetry describes itself as a “collection of tools, APIs, and SDKs [used] to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.”

We are sure that observability matters, particularly with DevOps, as modern micro-services- and container-based applications are too complex and too large to monitor manually. And open standards for telemetry are a very good idea, in part because they avoid vendor lock-on, but mainly because reported measurements mean the same thing across different companies and different vendors. Open Telemetry appears to be a mature standard, widely adopted, and effective.

So, why use a specialist Open Telemetry based tool to route your observability metrics to something like Splunk for analysis, rather than do it all with Splunk? Well, according to observIQ, you should use it because it is very efficient, close to the Open Source community, and cheaper.

We had an interesting chat with Mike Kelly, CEO of observIQ. We mostly asked technical questions, because the Observability plumbing matters. Even so, its products will be bought on the basis of the business outcomes they facilitate. One of its capabilities is the reduction of data by processing it at the edge, nearest to where it is generated, thus helping to reduce storage management overheads and minimising analytics latency. The business result of this could be increasing maturity with progress towards being a metrics-focused company.

I was a little disappointed that the idea of accessing mainframe data and integrating mainframe services with the rest of the IT estate didn’t generate more enthusiasm, but this certainly could be done, and mainframes are a source of a lot of valuable operating data (logs). I’d also like more information about using using AI/ML to support intelligent Observability, but it is early days yet.

On its website, observIQ claims extensive DevOps experience, with hundreds of metric and log integrations for monitoring cloud, hybrid cloud, on-prem infrastructure, It is trusted by many of the leading Cloud vendors and scales to the needs of the largest enterprises. Even so, you can start small and grow with success – the best approach (we think), especially in an area like Observability, where collecting data is not enough, it matters that you actually use what you have collected. And, if you are not used to operating with Observability oversight, remember that the sell has to be both top down and bottom up, which implies some investment in changing culture.

observIQ sees itself as evangelising a vendor-neutral basis for Observability, implemented higher up the stack. The recent announcement by Elastic, of the move to convergence of the Elastic Common Schema (ECS) and OpenTelemetry (OTel) Semantic Conventions into a single open schema that is maintained by OpenTelemetry, should be welcome to users of these two widely adopted specifications.

A particular success for observIQ is its adoption by Google. This is indeed impressive, but Google is hardly an average company. We need to raise some of the questions mentioned in this blog with more typical customers. Overall, however, we think that this company promises real benefits, for avoiding lock-in to vendor-specific Observability solutions.