BT

Twitter Open Sources R Programming Anomaly and Breakout Detection Packages

| by Alex Giamas Follow 9 Followers on Jan 31, 2015. Estimated reading time: 1 minute |

Twitter recently announced open sourcing an anomaly detection package in R. Anomaly detection is a major study field as it can denote different things. A major spike in followers or favorites around a topic can happen because something major is happening and this may be something that needs to be broadcast around the network. But this same spike can also happen because of bots and spammers, which means that to the contrary, action has to be taken to contain this activity.

Late last year, Twitter open sourced BreakoutDetection, an open source R package that makes breakout detection simple and fast. A breakout is characterized in this package by two steady states and an intermediate transition period. The transition between these two states may have occurred suddenly or gradually. This package can make sure that from a statistical standpoint and in the presence of anomalies, a breakout is sure to have occurred. Using the E-Divisive with Medians algorithm, the package can detect one or multiple breakouts in a given time series.

In contrast to breakout detection, anomaly detection is refetting to point-in-time anomalous data points. An anomaly can be global or local. A local anomaly is one that occurs inside a seasonal pattern, for example an extra five percent boost that can happen within the normal Christmas period boost that happens in activity. These are harder to detect than global anomalies, which typically extend above and beyond expected seasonality trends.
Another dimension in anomaly detection is positive and negative whereas positive anomaly may mean more tweets happening during Super Bowl and negative anomaly like less transactions per second may denote hardware and infrastructure issues.

The AnomalyDetection package works using the Generalized ESD test. It can detect both global and local, positive and negative anomalies. The code is available on GitHub.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT