Apache Spark has released version 1.3 of their project. The main improvements are the addition of the DataFrames API, better maturity of the Spark SQL, as well as a number of new methods added to the machine learning library MLlib, and better integration of Spark Streaming with Apache Kafka.
Matt Ranney, Chief Systems Architect at Uber, gave an overview of their dispatch system, responsible for matching Uber's drivers and riders. Ranney explained the driving forces that led to a rewrite of this system. He described the architectural principles that underpin it, several of the algorithms implemented and why Uber decided to design and implement their own RPC protocol.
At QCon London 2015 Phil Calcado shared lessons learnt from SoundCloud’s move from a monolithic to microservices architecture, and stated that the core requirements for building a microservice platform include developing capabilities for rapid provisioning, basic monitoring and rapid application deployment.
Luke Marsden stated at QCon London 2015, that although Docker containers provide a very useful deployment mechanism for development and test, the absence of host mobility for containers with state may provide problems in production deployments. The open source Flocker tool provides a mechanism to overcome this by allowing stateful containers to be moved between hosts.
At EclipseCon, the automated error reporting and UI freeze detecting tool - built into Eclipse Mars - was demonstrated. Having only been in the package for a short while, it has already helped identify and subsequently fix a number of problems. InfoQ spoke to the people behind the tools to find out more.
Google expanded its cloud portfolio to add a cheaper cloud-based backup service called Cloud Storage Nearline.
The fourth annual QCon New York will feature in-depth case studies from Netflix, Comcast, Goldman Sachs, NFL Digital Media, and others. Software Engineers, Technical Team Leads & Data Scientists at these companies will describe how they're dealing with some of the greatest challenges in the arenas of Microservices, Data Science, Reactive Architectures, Fraud Detection, DevOps & Software Delivery.
Ian Craggs presented about Paho, Mosquitto and MQTT/MQTT-SN at EclipseCon. InfoQ caught up with him and asked him what's happening in the MQTT space.
Today at EclipseCon, Mark Reinhold gave a keynote on "Java 9 and Beyond" which looked at the upcoming features in Java 9 and some of the future plans for the Java ecosystem. InfoQ summarises the main parts of what will and might happen in the future.
The Groovy team is joining the Apache Software Foundation (ASF). Guillaume Laforge, Groovy project lead, wrote about why they chose ASF over the Eclipse Foundation or the Software Conservancy foundation. To learn more about this announcement, InfoQ spoke to Mr. Laforge about the new direction.
After living with microservices for three years at Gilt we can see advantages in team ownership, boundaries defined by APIs and complex problems broken down. Challenges still exists in tooling, integration environments and monitoring, Yoni Goldberg explained in a presentation at the QCon London conference describing the challenges they encountered moving to a microservices architecture.
At the last QCon London, Michael Brunton-Spall, Technical Architect at the UK's Government Digital Service, expressed his views on how DevOps patterns are crucial to successfully operate microservices. Brunton-Spall identified the key ingredients to identify a microservice, explained how to build your first microservice and the necessary tools and practices to manage an ecosystem of microservices.
At QCon London 2015, Dave Farley proposed that although the state of software development has been suboptimal in the past, studies are revealing that the implementation of continuous delivery leads to considerable improvement. Farley stated that continuous delivery changes the economies of software development, and provides more rapid business idea validation and reduced defect rates.
When designing and building Halo 4, the next version in a video game series, a new solution was created based on the Actor model implemented by the Orleans framework. Caitie McCaffrey told in a presentation at the QCon London conference talking about the work designing and building the services supporting the new game.
Anna Shipman, technical architect at UK's Government Digital Service (GDS), revealed to the QCon London attendees how DevOps permeates their culture. GDS aims to lead the digital transformation of UK's government, "mak[ing] digital services and information simpler, clearer and faster". Its most well known site is GOV.UK, which provides government information and services.