Jesper Richter-Reichhelm, Head of Engineering at Wooga, spoke at GOTO Amsterdam 2014 about unexpected difficulties to find commoditized back end services that suited Wooga's mobile games needs. Combining reasonable cost, integration with an existing tool chain and transparent access to data proved impossible more often than not, leading to in-house development of said services.
Ayasdi announced last month a partnership with Cloudera, the biggest distributor of Apache Hadoop. The partnership that will ensure the compatibility of their solution with Cloudera Enterprise 5, the latest version of Cloudera’s big data platform based on Apache Hadoop.
DataTorrent is a real-time streaming and analyzing platform that can process over 1B real-time events/sec.
Splunk, a company specializing in searching, monitoring, and analyzing machine-generated data, has announced the release of Hunk 6.1. Hunk provides an analytics platform for big data. The new release also provides streaming resource libraries to connect Hunk to any NoSQL data store, such as Apache Cassandra, MongoDB, and Neo4j.
Recently, Spark graduated from the Apache incubator. Spark claims up to 100x speed improvements over Apache Hadoop over in-memory datasets and gracefully falling back to 10x speed improvement for on-disk performance. Based on Scala, it can run SQL queries and be used directly in R. It provides Machine Learning, Graph database capabilities and other further discussed in the article.
Hadoop is definitely the platform of choice for Big Data analysis and computation. While data Volume, Variety and Velocity increases, Hadoop as a batch processing framework cannot cope with the requirement for real time analytics. Spark, Storm and the Lambda Architecture can help bridge the gap between batch and event based processing.
Arun Kejariwal, from Twitter, talked at Velocity Conf London last month about forecasting algorithms used at Twitter to proactively predict system resource needs as well as business metrics such as number of users or tweets. Given the dynamic nature of their data stream, they found that a refined ARIMA model works well once data is cleansed, including removal of outliers.
Jeff Magnusson from Netflix team gave a presentation at QCon SF 2013 Conference about their Data Platform as a Service. Following up to this presentation, we will look at the technology stack and how it helps Netflix to tackle important business decisions.
The lean startup is a “scientific approach to creating and managing startups” as Eric Ries describes in the lean startup principles. It uses “hard things” like validated learning with experiments and data. But what the “soft things” like intuition, guts, feelings, passion, inspiration and fun, do they also matter when you are developing new products?
Thoughtworks recently released a new installment of their technology radar highlighting techniques enabling infrastructure as code, perimeterless enterprises, applying proven practices to areas without, and lightweight analytics.
DNN Social enables customers to interact with the site interface via blogs, discussion forums, FAQ's and includes features such as gamification, analytics, ideation and activity stream, which enables site administrators to gauge the effectiveness of interaction.
ThoughtWorks's latest "Technology Radar" focuses on mobile, accessible analytics, simple architectures, reproducible environments, and data persistence done right.
Precog has recently announced a Big Data warehousing and analysis service which takes care of the data capture, storage, transformation, analysis and visualization process and the infrastructure on which it runs, but leaving open various access points throughout the service via RESTful APIs enabling developers and data scientists to control the entire process.
Oracle Big Data Appliance and Big Data Connectors support integration with Hadoop, Cloudera Manager and Oracle NoSQL Database. Oracle announced last month the availability of Big Data Appliance and Connectors as well as partnership with Cloudera. They also recently announced the Advanced Analytics for Big Data by integrating R statistical programming language into Oracle Database 11g.
In a recent press news from 13th December, SAP announced at the SAP Influencer Summit in Boston that “leading software vendors are adopting the open SAP HANA platform for their existing products and building completely new applications.” Among them are companies such as T-Mobile and TIBCO.