Real-time analysis of event streams has a new focus in Big Data platforms, both on-premise and in the cloud. AWS have released Amazon Kinesis Analytics, a rival to Azure StreamAnalytics. Both platforms use a simple SQL language for complex querying, and move Big Data analysis into a SaaS-like space.
A team of scientists at IBM Research in Zurich, have created an artificial version of neurons using phase-change materials to store and process data. These phase change based artificial neurons can be used to detect patterns and discover correlations in Big Data (real-time streams of event based data) and unsupervised machine learning at high speeds using very little energy.
Data science is about the data that you need; deciding which data to collect, create, or keep is fundamental argues Lukas Vermeer, an experienced Data Science professional and Product Owner for Experimentation at Booking.com. True innovation starts with asking big questions, then it becomes apparent which data is needed to find the answers you seek.
On July 15th, Microsoft announced the Azure Premium Messaging service has reached General Availability (GA). Premium Messaging targets customers who would like more predictable messaging performance. InfoQ reached out to Dan Rosanova, Principal Program Manager on the Azure Service Bus team for additional insight into this milestone.
InfoQ's Rags Srinivas talks to Basho's CTO Dave McCrory about the open sourcing of Riak TS 1.3 which is geared to handle time series data.
Netflix's goal is to predict what you want to watch before you watch it. They do this by running a number of machine learning (ML) workflows every day. Meson is a workflow orchestration and scheduling framework that manages the lifecycle of all these machine learning pipelines that build, train and validate personalization algorithms to help with the video recommendations.
A full snapshot of more than 2.8 million open source project hosted on GitHub is now available in Google’s BigQuery, Google and GitHub announced. This will make it possible to query almost 2 billion source files hosted on GitHub using SQL.
In her presentation "Large-Scale Stream Processing with Apache Kafka" at QCon New York 2016, Neha Narkhede introduces Kafka Streams, a new feature of Kafka for processing streaming data. According to Narkhede stream processing has become popular because unbounded datasets can be found in many places. It is no longer a niche problem like, for example, machine learning.
LinkedIn’s Joel Koshy details their Kafka usage, debugging and monitoring two production incidents in using the core Kafka infrastructure concepts, semantics and behavioral patterns to plan for and detect similar problems in the future.
LinkedIn recently detailed open-sourced Kafka Monitor service that they're using to monitor production Kafka clusters as well as extensive testing automation, leading them to identify bugs in the main Kafka trunk and contribute solutions to the open-source community.
Confluent Platform 3.0 messaging system from Confluent, the company behind Apache Kafka messaging framework, supports Kafka Streams for real-time data processing. The company announced last week the general availability of the latest version of the open source Confluent platform.
Cloudera announced their partnership with MIT & Harvard's Broad Institute and detailed some of their experience with the Genome Analytics Toolkit pipeline.
Two years after the first release of Apache Spark, Databricks announced the technical preview of Apache Spark 2.0 , based on upstream branch 2.0.0-preview. The preview is not ready for production, neither in terms of stability nor API, but is a release intended to gather feedback from the community ahead of the general availability of the release.
Amazon has recently announced an update to their Amazon Kinesis Service. In this update, three new features have been added to Amazon Kinesis Streams and Amazon Kinesis Firehose including support for Elasticsearch Service Integration, Shard-Level Metrics and Time-Based Iterators.
AWS engineers Christopher Crosbie and Ujjwal Ratan detail using Spark on EMR for precision medicine data analysis on the ADAM platform with data from the 1000 genomes project.