InfoQ Homepage Distributed Systems Content on InfoQ
-
Dynein – an Asynchronous Background Job Service from Airbnb
At Airbnb, they move time consuming, resource intensive tasks over to asynchronous background jobs to improve scalability. The job scheduling system has become a very important component and they have therefore built Dynein, a distributed delayed job queueing service and scheduler. In a blog post, Andy Fang from Airbnb describes the background and challenges in designing and building the service.
-
Experience Running Spotify’s Event Delivery System in the Cloud
Event delivery is a key component at Spotify; the events contain important data about users, actions they take, and operational logs. After running the event delivery system in the cloud for 2 ½ years, Bartosz Janota and Robert Stephenson have written a blog post discussing what they have achieved and how they have been able to evolve and simplify the system by moving up the stack in the cloud.
-
Mind Your State for Your State of Mind: Pat Helland at QCon SF
The features of different types of data storage should be considered when selecting how data is stored in a system. Is always reading correct data, or low latency, most important? In his keynote at this year’s QCon San Francisco, Pat Helland described trends in storage and computing, durable and session state semantics, and other aspects of storage like transactions, identity and immutability.
-
AWS IoT Day Recap: Eight New Powerful Features
As part of AWS re:Invent pre-event announcements, Amazon shared eight new features available within their Internet of Things platform. These new features include: secure tunneling, configurable endpoints, custom domains for configurable endpoints, enhanced custom authorizers, fleet provisioning, Alexa Voice Services (AVS) integration and AWS IoT Greengrass enhancements.
-
Oracle Expands Cloud Native Services, Adds Kafka Streaming, API Gateway and Logging Support
In a recent blog post, Oracle announced the limited availability of three news service offerings in its Oracle Cloud Native Services platform. The three new services include Kafka Compatibility for Oracle Streaming, an API Gateway for managing connectivity to serverless components and containers and a Logging service that supports log management and analytics across resources and applications.
-
Jay Kreps: Events, Event Streams and Their Importance in a Digital Business
Organizations are moving more and more processes into software, Jay Kreps notes in a blog post, and adds that in an accompanying change businesses are increasingly defined in software – the core processes are specified and executed in software. To support this transition, he believes we have to move away from traditional databases into working with the concepts of events and events streams.
-
Gutenberg – a Publish-Subscribe Service for Datasets Created by Netflix
To propagate datasets from a single producer to multiple consumers, Netflix has created Gutenberg, a service using a publish-subscribe technique to propagate versioned datasets between their microservices. In a blog post, Ammar Khaku, senior software engineer at Netflix, describes an overview of the design as well and some use cases for Gutenberg.
-
Extending the Reach of SQL to IoT Microcontrollers, ITTIA and Cypress Release SDK
In a recent press release, ITTIA, a maker of embedded database software for Internet of Things (IoT) devices, and Cypress Semiconductor Corp, announced a collaborative IoT device and data management capability. The new capability integrates SQL into the WICED SDK and unlocks the power of flash media on Cypress wireless microcontrollers (MCU).
-
Minimizing Backend Complexity with Dark: A New Language with Integrated Editor and Infrastructure
Dark aims to simplify the development of backends by minimizing complexity. Dark is a programming language with an integrated editor and infrastructure for developing and delivering backend applications.
-
Improving Blockchain Performance Off-Chain, Hyperledger Announces Avalon
In a recent blog post, the Hyperledger project announced a new project, called Hyperledger Avalon, that addresses some of the scalability and privacy challenges that are currently associated with many blockchain projects. The projects seek to address these scalability and privacy challenges through the use of trusted off-chain processing, while ensuring the transactions are secure and resilient.
-
Sense and Nonsense in Event Thinking and Microservices
Modularity in the systems we are building is very important, but there are anti-modularity forces that we must deal with to be able to achieve this modularity. In a presentation at the recent Event-driven Microservices Conference, held by AxonIQ, Allard Buijze shared his thoughts and experience building systems based on DDD, CQRS, microservices and event sourcing.
-
Day Two Problems When Using CQRS and Event Sourcing
There are a lot of good reasons for building a CQRS and event-sourcing based system, but there are also problems that appear only after an application is in production. In a presentation at the recent Event-driven Microservices Conference held by AxonIQ, Joris Kuipers shared his experience running and evolving CQRS and event sourced applications in production.
-
Google Releases Cloud Dataproc for Kubernetes in Alpha
Google Cloud Dataproc is an open-source data and analytic processing service based on Hadoop and Spark. Google has recently announced the alpha availability of Cloud Dataproc for Kubernetes, which provides customers with a more efficient method to process data across platforms.
-
ApacheCon 2019 Keynote: Google Cloud Enhances Big-Data Processing with Kubernetes
At ApacheCon North America, Christopher Crosbie gave a keynote talk title "Yet Another Resource Negotiator for Big Data? How Google Cloud is Enhancing Data Lake Processing with Kubernetes." He highlighted Google's efforts to make Apache big-data software "cloud native" by developing open-source Kubernetes Operators to provide control planes for running Apache software in a Kubernetes cluster.
-
Reactive Foundation Launched under the Linux Foundation
The Linux Foundation announced the launch of the Reactive Foundation, a community of leaders established to accelerate technologies for building the next generation of networked applications. The foundation is made up of Alibaba, Facebook, Lightbend, Netifi and Pivotal as initial members, and includes the successful open source Reactive Streams and RSocket specifications.