Last week, Hashicorp released version 0.7 of Consul its open-source distributed service discovery and configuration tool. Tagged a "very large release", it introduces transactions for key/value updates, replication of ACLs across datacenters, improvements to its Raft and Gossip protocol implementations and optimisation of corresponding timings.
Real-time analysis of event streams has a new focus in Big Data platforms, both on-premise and in the cloud. AWS have released Amazon Kinesis Analytics, a rival to Azure StreamAnalytics. Both platforms use a simple SQL language for complex querying, and move Big Data analysis into a SaaS-like space.
A team of scientists at IBM Research in Zurich, have created an artificial version of neurons using phase-change materials to store and process data. These phase change based artificial neurons can be used to detect patterns and discover correlations in Big Data (real-time streams of event based data) and unsupervised machine learning at high speeds using very little energy.
Data science is about the data that you need; deciding which data to collect, create, or keep is fundamental argues Lukas Vermeer, an experienced Data Science professional and Product Owner for Experimentation at Booking.com. True innovation starts with asking big questions, then it becomes apparent which data is needed to find the answers you seek.
On July 27th, Microsoft announced their Integration Platform as a Service (iPaaS) offering, Logic Apps has reached General Availability (GA). The GA release includes additional management support, telemetry events, alerts, and consumption-based pricing. InfoQ reached out Jim Harrer, principal group program manager at Microsoft to gain further insight into this Logic Apps release.
On July 15th, Microsoft announced the Azure Premium Messaging service has reached General Availability (GA). Premium Messaging targets customers who would like more predictable messaging performance. InfoQ reached out to Dan Rosanova, Principal Program Manager on the Azure Service Bus team for additional insight into this milestone.
InfoQ's Rags Srinivas talks to Basho's CTO Dave McCrory about the open sourcing of Riak TS 1.3 which is geared to handle time series data.
Netflix's goal is to predict what you want to watch before you watch it. They do this by running a number of machine learning (ML) workflows every day. Meson is a workflow orchestration and scheduling framework that manages the lifecycle of all these machine learning pipelines that build, train and validate personalization algorithms to help with the video recommendations.
A full snapshot of more than 2.8 million open source project hosted on GitHub is now available in Google’s BigQuery, Google and GitHub announced. This will make it possible to query almost 2 billion source files hosted on GitHub using SQL.
At DockerCon, Docker released version 1.12 of the core product, Docker Engine. The biggest new feature is that Docker Swarm is no longer a separate tool - now it's built into Docker Engine, making it easier to combine multiple Docker hosts into a single logical unit for increased scale and reliability.
At DockerCon 2016, held in Seattle, the latest 1.12 beta version of Docker Engine was announced that includes the integration of Docker Swarm to provide container orchestration. Additional announcement included: the Docker for Mac and Windows has now been made public; a private beta for Docker for AWS and Azure has been opened; and the release of a 'DAB' file format for packaging artifacts.
In her presentation "Large-Scale Stream Processing with Apache Kafka" at QCon New York 2016, Neha Narkhede introduces Kafka Streams, a new feature of Kafka for processing streaming data. According to Narkhede stream processing has become popular because unbounded datasets can be found in many places. It is no longer a niche problem like, for example, machine learning.
LinkedIn’s Joel Koshy details their Kafka usage, debugging and monitoring two production incidents in using the core Kafka infrastructure concepts, semantics and behavioral patterns to plan for and detect similar problems in the future.
Moving applications to the cloud has somewhat become commodity in the meantime - not only for big players, but also for smaller companies that rely on flexibility and resource utilization. In his presentation "Implementing Infrastructure as Code", Kief Morris, cloud practice lead at ThoughWorks, shares some key principles and recommendations on how to leverage cloud based infrastructure.
LinkedIn recently detailed open-sourced Kafka Monitor service that they're using to monitor production Kafka clusters as well as extensive testing automation, leading them to identify bugs in the main Kafka trunk and contribute solutions to the open-source community.