Yahoo! has benchmarked three of the main stream processing frameworks: Apache Flink, Spark and Storm.
After several years of development, MBrace 1.0 was released last week. MBrace is a programming model for scalable cloud data scripting and programming with F# and C#. The project consists mainly of code libraries and cloud providers runtime.
GameAnalytics, maker of a free analytics platform, has recently open sourced gascheduler an Erlang library that provides a generic scheduler for parallel execution of distributed tasks. InfoQ has spoken to Chris de Vries, one of gascheduler’s creators.
ELIoT (Extensible Language for the Internet of Things) is a simple and small programming language aiming to make distributed programming easier. A program in ELIoT may appear as a sigle program, but it actually runs on different computers, so, e.g., a variable or function declared on one computer is transparently used on another.
Twitter has replaced Storm with Heron which provides up to 14 times more throughput and up to 10 times less latency on a word count topology, and helped them reduce the needed hardware to a third.
Martin Thompson answers a few questions about the opportunity for developers and architects to introduce custom protocols to their system's interaction points.
Hadoop is definitely the platform of choice for Big Data analysis and computation. While data Volume, Variety and Velocity increases, Hadoop as a batch processing framework cannot cope with the requirement for real time analytics. Spark, Storm and the Lambda Architecture can help bridge the gap between batch and event based processing.
Mobile Backend as a Service provider AnyPresence continues to hone their chops. Launching the fifth update to their self-titled platform geared for the enterprise. Co-founder Rich Mendis provides some insights for InfoQ readers…
Twitter has open-sourced Storm, its distributed, fault-tolerant, real-time computation system, at GitHub under the Eclipse Public License 1.0. Storm is the real-time processing system developed by BackType, which is now under the Twitter umbrella.
FlightCaster recently open sourced Crane, a tool for distributing and remotely controlling Clojure instances, currently specialized for EC2. Incanter is a Clojure library and tool that makes R-like statistical computations easy with Clojure. Also: the build and dependency management tool Leiningen 1.0 is now available.
With the multiplicity of existing remoting mechanisms it is often necessary to build clients in a way that allows to swap/introduce new protocols with no/minimal impact to the client’s implementation. A new framework – CRISPY - provides support for such implementations.
Tim Bray of Sun Microsystems writes of the Fallacies of Distributed Computing; He observes that despite its profound implications when designing distributed systems, “you don’t often find them coming up in conversations about building big networked systems”.
Recently, an early release draft of a Distributed OSGi requirements and design document has been published, along with a reference implementation as part of Apache CXF. In a new article, Eric Newcomer writes about the current status of distributed OSGi and explains the reasons for standardizing it in the first place, and its significance to the OSGi specification and community.
Google caused a stir by releasing Protocol Buffers, a binary serialization format. We take a look at what exactly Protocol Buffers are and what alternatives are available in ASN.1 or Facebook's Thrift.
In this interview from QCon San Francisco 2007, Randy Shoup discusses the architecture of eBay. Topics discussed include eBay's architectural principles, horizontal and vertical partitioning, ACID vs. BASE, handling data inconsistency, distributed caching, updating eBay on the fly, architectural and coding standards, eBay's search infrastructure, grid computing, and SOA.