A full snapshot of more than 2.8 million open source project hosted on GitHub is now available in Google’s BigQuery, Google and GitHub announced. This will make it possible to query almost 2 billion source files hosted on GitHub using SQL.
At DockerCon, Docker released version 1.12 of the core product, Docker Engine. The biggest new feature is that Docker Swarm is no longer a separate tool - now it's built into Docker Engine, making it easier to combine multiple Docker hosts into a single logical unit for increased scale and reliability.
At DockerCon 2016, held in Seattle, the latest 1.12 beta version of Docker Engine was announced that includes the integration of Docker Swarm to provide container orchestration. Additional announcement included: the Docker for Mac and Windows has now been made public; a private beta for Docker for AWS and Azure has been opened; and the release of a 'DAB' file format for packaging artifacts.
In her presentation "Large-Scale Stream Processing with Apache Kafka" at QCon New York 2016, Neha Narkhede introduces Kafka Streams, a new feature of Kafka for processing streaming data. According to Narkhede stream processing has become popular because unbounded datasets can be found in many places. It is no longer a niche problem like, for example, machine learning.
LinkedIn’s Joel Koshy details their Kafka usage, debugging and monitoring two production incidents in using the core Kafka infrastructure concepts, semantics and behavioral patterns to plan for and detect similar problems in the future.
Moving applications to the cloud has somewhat become commodity in the meantime - not only for big players, but also for smaller companies that rely on flexibility and resource utilization. In his presentation "Implementing Infrastructure as Code", Kief Morris, cloud practice lead at ThoughWorks, shares some key principles and recommendations on how to leverage cloud based infrastructure.
LinkedIn recently detailed open-sourced Kafka Monitor service that they're using to monitor production Kafka clusters as well as extensive testing automation, leading them to identify bugs in the main Kafka trunk and contribute solutions to the open-source community.
As part of the ongoing transition to the module system, CORBA and other Java EE modules won't be included in the default classpath from Java 9 onwards. These modules will still be available, but specific command line flags will have to be used to be able to use them. The change will only affect non-modular applications targeting Java 9, for modular ones already need to indicate their dependencies.
Confluent Platform 3.0 messaging system from Confluent, the company behind Apache Kafka messaging framework, supports Kafka Streams for real-time data processing. The company announced last week the general availability of the latest version of the open source Confluent platform.
Cloudera announced their partnership with MIT & Harvard's Broad Institute and detailed some of their experience with the Genome Analytics Toolkit pipeline.
Two years after the first release of Apache Spark, Databricks announced the technical preview of Apache Spark 2.0 , based on upstream branch 2.0.0-preview. The preview is not ready for production, neither in terms of stability nor API, but is a release intended to gather feedback from the community ahead of the general availability of the release.
Amazon has recently announced an update to their Amazon Kinesis Service. In this update, three new features have been added to Amazon Kinesis Streams and Amazon Kinesis Firehose including support for Elasticsearch Service Integration, Shard-Level Metrics and Time-Based Iterators.
AWS engineers Christopher Crosbie and Ujjwal Ratan detail using Spark on EMR for precision medicine data analysis on the ADAM platform with data from the 1000 genomes project.
Supergiant is a container hosting platform built using Kubernetes for distributed, stateful applications.
Summary of DevOps Days Kiel day 1 talks.