Grafana Labs, the creator of the open-source metrics platform Grafana, recently released the distributed tracing backend Grafana Tempo. It only requires object storage like Amazon S3 or Google Cloud Storage (GCS) to operate, in comparison with similar solutions that typically require the use of one or more data stores, such as ElasticSearch or Cassandra.
By using end-to-end latency graphs (traces) in near real-time, distributed tracing systems help engineers understand the reason why certain requests take longer than others. With a retention period of 14 days, Tempo currently consumes 170k spans (individual operations) / second, groups these, and stores them in GCS.
Joe Elliott, a senior backend engineer at Grafana Labs, made the announcement, describing challenges with Grafana engineers around the need to find a specific trace.
With the requirement of 100% sampling and fewer efforts to maintain the Elasticsearch or Cassandra cluster required, engineers at Grafana were unhappy with their current downsampled distributed tracing system.
Enabling a searchable index for trace IDs, Grafana Tempo integrates with any existing logging system to create links from trace IDs in log lines. Teams can search by path, status code, latency, user, or IP onto the same log line as a trace id. Elliott also briefly referred to version 2.0 of Loki, Grafana's log aggregation system.
Source: https://grafana.com/blog/2020/10/27/announcing-grafana-tempo-a-massively-scalable-distributed-tracing-system/
He further introduced Exemplars, which illustrate the trace pattern, which has links to traces based on the metrics query directly embedded in the Grafana graph. Currently in development, Exemplars is planned to have support for Grafana and Prometheus.
He also described Grafana Cloud Agent, a data collector capable of sending metrics and log data to Grafana Cloud, enabling the operator to reach the logs from exemplar with ease.
Source: https://grafana.com/blog/2020/10/27/announcing-grafana-tempo-a-massively-scalable-distributed-tracing-system/
The surrounding community has grown, providing feedback on the release as Tempo reached 1000 stars on Github. Twitter user Wiard van Rij tweeted, "I've made a proof of concept with @grafana new tool Tempo for tracing. Honest opinion is that the idea and tool are really superb. Yet I would consider it pre-pre-alfa. Which is cool. It's open source and with proper adoption and support this will change!" Another Twitter user Vale@ganbara.nai tweeted, "I just love the ease of use of grafana-prometheus-loki so I'm looking forward to what tempo has as well."
The team at Grafana Labs has been providing the community with the steps to configure and performance improvements. Robert Fratto, a senior software engineer at Grafana Labs, blogged about configuring the Grafana Cloud Agent to collect traces and send them to Tempo. Elliott also tweeted about the performance improvement related to Bloom filter sharding.
For further details, readers can watch the on-demand ObservabilityCon session, Tracing made simple with Grafana, and join the conversation in Grafana Slack #tempo channel or the tempo-users google group. The documentation for Tempo is available in this link.