BT

Logz.io Offers Machine Learning Based Log Analysis

by Hrishikesh Barua on Nov 15, 2016 |

Logz.io offers a hosted service which performs intelligent log analysis by using machine learning to derive insights from human interactions with log data that include discussions on tech forums and public code repositories.

Modern applications churn out logs and metrics at a huge rate. DevOps teams utilize tools like the ELK stack to centralize the log aggregation and indexing process. However, the sheer amount of logs is not easy to wade through. The amount of metrics collected has also grown tremendously so that traditional monitoring tools cannot do much more than display it. This increase in data volume has also led to a situation where thresholds-based alerting is not viable, thus leading to the rise of anomaly detection systems like Anomaly.io, Datadog and SumoLogic, and also open source ones like Etsy’s Skyline.

Logz.io takes this further by offering insights into error conditions, which are derived by using machine learning on data gathered from human interactions on the internet. These interactions include discussions about similar error conditions on sites like StackOverflow and Github. Since a simple Google search for most error conditions usually throws up many results, how does Logz.io filter out the relevant ones? InfoQ got in touch with Tomer Levy, CEO and Co-Founder at Logz.io, to find out more:

Logz.io uses machine learning to understand the context of the thread, the product to which the thread is referring, whether there was a resolution, and if it was satisfactory to the person asking the question. We also cross-reference these threads with data we’ve gathered from our own users.


In addition, Levy says, “after we bring an insight to a user’s attention, we assess the relevance based on his or her interaction with the insight.“ This completes a feedback loop so that the system can improve with time.

Searching for an error or exception for a product on the internet would lead to results in most cases since most modern products are built using open source libraries. In the rare cases that a product is built completely from scratch without using any publicly available code, Logz.io plans to release “private insights” in the future that will “enable organizations to define their own exceptions and the reasons behind them”, according to Levy.

Logz.io uses the Elasticsearch-Logstash-Kibana (ELK) stack under the hood. It’s a service hosted on Logz.io’s own cloud, which means that log data has to be piped there from the application. In cases where the organization runs its own ELK stack, Logz.io can integrate with the log shippers that reside on-premises and help securely transfer the data to the Logz.io cloud.

In response to how they address enterprise concerns about data security, Levy said:

We’ve invested in security from data encryption to comprehensive security processes. Logz.io has hired Earnest & Young to audit its security infrastrcture and process and was recently passed their audit for SOC-2 Type 2 and ISO-27001 compliance. In addition, the company hosts data in the United States, and EU-based companies can choose to store data in Frankfurt to ensure compliance with EU regulations.

Rate this Article

Relevance
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss
General Feedback
Bugs
Advertising
Editorial
Marketing
InfoQ.com and all content copyright © 2006-2016 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT

We notice you're using an ad blocker

We understand why you use ad blockers. However to keep InfoQ free we need your support. InfoQ will not provide your data to third parties without individual opt-in consent. We only work with advertisers relevant to our readers. Please consider whitelisting us.