InfluxData releases into general availability the new version of its database engine called InfluxDB IOx. It is now available to be used in InfluxDB Cloud.
The new engine, developed in Rust, as said in an official InfluxData tweet, can handle metrics, events, and traces. Metrics are time series polled at regular intervals from the sources and are useful to understand the performances and the availability of a service. Events are state changes triggered by some conditions (i.e. deployed code, HTTP 5XX errors) and can be correlated to the metrics to have additional insights. Traces capture information to show the request propagation in a distributed system.
InfluxDB IOx allows InfluxDB Cloud to ingest the data in real-time and derive metrics on the fly. InfluxDB can now better handle the use cases of observability and distributed tracing, which rely on high cardinality data. In a distributed system it is important to know how each component is working in relation to the others, as some components depend on others, so errors, bottlenecks, and delays can impact the performance of the entire system.
Tracing is an observability concept used to understand how the different pieces work together. A trace provides a view of a request, operation, task, or another unit of work as it works in the distributed system. Tracing data are, by definition, high-cardinality data. For a time series database, cardinality can become a problem for data with unbounded values such as user IDs, IP addresses, and container IDs. High cardinality can affect performances at scale due to how the database index data.
The new engine also has a new columnar structure and compression abilities, which allows InfluxDB Cloud to handle tag values of unbounded data without a drop in performance. The new format for data persistence is Apache Parquet, which allows better compression (and low storage cost). When new data arrives, IOx writes data to columns of the table and saves it to a new Parquet file. When IOx writes data, it includes some hints in the Parquet metadata to describe the column context, these metadata are used by the query engine at query time to skip over entire Parquet files and/or not interesting portions of files. Parallelism, in-memory cache, and pushdown concepts are also used by the query engine to improve low-latency query performances.
With the utilization of Apache DataFusion, a Rust-based query engine, developers can now use native SQL queries to gather information. They can still use Flux language for more advanced data processing.
At the time of writing, the new database engine is available in two AWS regions Virginia (us-east-1) and Frankfurt (eu-central-1), but new availability zones are planned, as well as the adding availability for Azure and Google Cloud.