Matt Ranney, Chief Systems Architect at Uber, gave an overview of their dispatch system, responsible for matching Uber's drivers and riders. Ranney explained the driving forces that led to a rewrite of this system. He described the architectural principles that underpin it, several of the algorithms implemented and why Uber decided to design and implement their own RPC protocol.
The goal of software is to sustainably minimize lead time to positive business impact, everything else is detail, Dan North claimed in a presentation at the QCon London conference describing ways of reasoning about code and how this leads him into an architecture style that may fit microservices.
Pivotal recently released Spring XD 1.1 GA with new features including stream processing with Reactor, RxJava, Spark Streaming and Python. Additionally support for Kafka, batching and compression with RabbitMQ, and support for container group management when running on YARN are now featured.
The assumption that a large system must have a single environment, often with a one-to-one mapping between a project’s scope and the system built are challenged today Stefan Tilkov explains when looking into ways to split a large system into smaller parts and comparing the characteristics of systems, applications and microservices.
Google announced last week the release of open source MapReduce framework for C, called MR4C, that allows developers to run native code in Hadoop framework. MR4C framework brings together the performance and flexibility of natively developed algorithms with the scalability and throughput provided by Hadoop execution framework.
Pivotal has decided to open source core components of their Big Data Suite and has announced the Open Data Platform, an initiative promoting open source and standardization for Big Data.
Project Pachyderm Aims to Build "Modern" Hadoop using Docker and CoreOS.
An article by Jin Scott - A tale of two clusters: Mesos and YARN – describes hardware silos created by using different resource managers on different hardware clusters, most popular being Mesos and Yarn and introduces Myriad – a solution allowing to run a YARN cluster on Mesos.
Apache Hive has released version 1.0 of their project on February 6th, 2015. Originally planned as version 0.14.1, the community voted to change the version numbering to 1.0.0 to reflect the amount of maturity the project has reached.
CoreOS announced the availability of etcd 2.0, the first stable version of the open source distributed key-value store.
When adopting a microservices architecture, using an external architect to create the design of a service instead of helping a team make their own decisions about design and implementation is one of several traps or bad practices that Vladimir Khorikov has experienced in his work.
Amazon recently announced EMRFS, an implementation of HDFS that allows EMR clusters to use S3 with a stronger consistency model. When enabled, this new feature keeps track of operations performed on S3 and provides list consistency, delete consistency and read-after-write-consistency, for any cluster created with Amazon Machine Image (AMI) version 3.2.1 or greater.
Reasons for building microservices are often about using isolation as a means to handle change. Sharing code between services couples your services to each other reducing the effectiveness of the isolation and the ability to handle change, David Dawson writes in a series of blog posts questioning the Don’t Repeat Yourself (DRY) principle in connection with microservices.
Five years ago many NoSQL databases were pre version 1.0 and when, it came to the CAP tradeoff, choosing availability over consistency was in vogue. Fast forward to today and distributed, fault tolerant transactions are moving into the fore as a new round of NoSQL databases seek to redefine our NoSQL expectations.
Splice Machine version 1.0 supports analytic window functions and integration with Hadoop ecosystem. Splice Machine team recently released their Hadoop based RDBMS data management solution that can be used for transactional workloads on Hadoop.