BT

New York Times Lab Introduces Visual Stream Processing Tool

by Michael Hausenblas on Apr 01, 2014 |

The New York Times R&D Lab has released streamtools, a general purpose, graphical tool for dealing with streams of data.

Mike Dewar, the streamtools project lead at NY Times Lab describes the motivation as follows:

Over the last 20 years we have invested heavily in tools that deal with tabulated data, from Excel, MySQL, and MATLAB to Hadoop, R, and Python+Numpy. These tools, when faced with a stream of never-ending data, fall short and diminish our creative potential. In response to this shortfall we have created streamtools.
Streamtools provides a GUI that runs in a browser, allowing users to explore, analyse, modify and learn from streams of data. The vocabulary of operations streamtools defines is straightforward: data flows from blocks through connections to other blocks. These can be connected together to create live data processing systems without the need for programming or complicated infrastructure:
  • A block perfoms some operation on each message it recieves, and that operation is defined by the block's type.
  • Each block has zero or more rules which define that block's behaviour.
  • Each block has a set of named routes that can recieve data, emit data, or respond to queries.
  • Blocks can be connect via routes, using connections. 
  • A collection of connected blocks is called a pattern, and it is possible to export and import whole patterns from a running instance of streamtools via a JSON formatted descriptor document.
Streamtools is written in Go, Google's increasingly popular programming language and is available via GitHub under the Apache 2 license. Dewar explains the decision for Go
We chose Go because it lets us write code that's very close to the idiom we're trying to present to the user. So every block is its own goroutine, every connection is a pair of channels. It makes for a very straightforward abstraction, which we hope will let people understand the systems they're building. This also means that writing new blocks is really simple, which we hope will encourage the community to make the blocks they find useful. Go also allows us to write safe, performant code, which is great for our day-to-day work at The Times.
The streamtools team at NY Times Lab envisions that stream-based APIs are becoming much more prevalent and that using streams changes how we draw conclusions about the world.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Usecases for the tool by Aravind Yarram

What are the usecases of this tool? Can't figure out where to use this

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

1 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT