This article discusses what stream processing is, how it fits into a big data architecture with Hadoop and a data warehouse (DWH), when stream processing makes sense, and what technologies and products you can choose from.
GridGain recently announced the In-Memory Accelerator for Hadoop, offering the benefits of in-memory computing to Hadoop based applications. It includes two components: an in-memory file system and a MapReduce implementation. InfoQ spoke with Nikita Ivanov, CTO of GridGain about the architecture of the product.
Spring XD (eXtreme Data) is Pivotal’s Big Data play. It joins Spring Boot and Grails as part of the execution portion of the Spring IO platform. Whilst Spring XD makes use of a number of existing Spring projects it is a runtime environment rather than a library or framework, comprising a bin directory with servers that you start up and interact with via a shell.
The MLConf conference was going strong in NYC on April 11th and was a full day packed with talks around Machine Learning and Big Data, featuring speakers from many prominent companies.
Lambda Architecture proposes a simpler, elegant paradigm designed to process large amounts of data. In this article, author discusses Lambda Architecture with the help of a sample Java application. 6
This article provides an overview of tools and libraries available for embedded data analytics & statistics, both stand-alone software packages and programming languages with statistical capabilities.
In this article, authors discuss the role of big data and Hadoop in security analytics space and how to use MapReduce to process data for security analysis.
How to use various tools such as Apache Avro, Apache Crunch, Cloudera ML and the Cloudera Development Kit to build applications that use Hadoop.
Raffi Krikorian, Vice President of Platform Engineering at Twitter, gives an insight on how Twitter prepares for unexpected traffic peaks and how system architecture is designed to support failure. 1
Jon Natkins explains in this article how to create a personalized recommendation system fed with large amounts of real-time data using Kiji, which leverages HBase, Avro, Map-Reduce and Scalding.
How do you bringing agility into big data? Learn what makes analytics uniquely different than application development, and how to adapt agile principles and practices to the nuances of analytics.
Elasticsearch is an open source, distributed real-time search and analytics engine for the cloud. InfoQ spoke with Costin Leau about Elasticsearch and how it integrates with Hadoop and Big Data.