BT

Your opinion matters! Please fill in the InfoQ Survey!

Apache Metron Graduates to Top-Level Project

| by Dylan Raithel Follow 4 Followers on May 18, 2017. Estimated reading time: 2 minutes |

Hortonworks and Apache announced Metron graduating to a top-level project. Metron is the latest evolution of an all-in-one security telemetry data capture, streaming analytics and response platform whose lineage started at Cisco with the OpenSOC project, an open-source security framework for big data systems. Metron provides log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment and applies current threat intelligence information to security telemetry, and does so with a single platform.

Conceptually, Metron is comprised of four components: Data capture and ingest, real-time processing, guaranteed data-persistence and storage, and machine learning models as a service that drive monitoring and alerts around risk.

Metron is at its core a Kappa architecture, a variant of Lambda architecture implemented with Apache Kafka as its unified data bus, and Apache Storm as the processing component. A Bro plugin provides the ability to forward Bro logs to Kafka. This lets Metron capture data that are specifically useful for deep packet inspection, capture and reconstruction while taking advantage of Kafka's guarantees and integration with the rest of the big data ecosystem.

For data capture, telemetry data can be posted to Metron's message-bus, and persisted or processed in real-time via Storm to HBase. Once the data is captured a number of options are available for search indexing and real-time, and near real-time processing. Metron provides interfaces to optionally interface HBase with ElasticSearch, or Lucene and Solr. Default management and dashboarding interfaces are built on Kibana.

There are a few features that make Metron different from an emerging standard in data pipelines. First is its integration with a set of data transformation utilities and API's via Stellar, a threat intelligence triage and field transformations language, that operate as functions deployed and executed via Metron's RESTful modeling-as-a-service (MaaS). MaaS functions are managed via Yarn and are designed for implementing as real-time or near-real-time threat detection and response mechanisms. A data-enrichment tool set provides the ability to manage and load various enrichment and threat intelligence sources into Metron's HBase data sink. The machine learning models deployed through the MaaS are meant to augment the behavior of this data enrichment step. A set of profilers. Lastly, a profiler mechanism executes feature extraction and windowing over the real-time and near-real-time telemetry data as it's coming in off the message bus, and as it gets persisted to HBase.

Owen O'Malley is the lead on the project at Apache and is responsible for initially porting Metron over from OpenSOC.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT