InfoQ Homepage Queue Content on InfoQ
News
RSS Feed-
Netflix Builds a Custom High-Throughput Priority Queue Backed by Redis, Kafka and Elasticsearch
Netflix recently published how it built Timestone, a custom high-throughput, low-latency priority queueing system. They built it using open-source components such as Redis, Apache Kafka, Apache Flink and Elasticsearch. Engineers state that they made Timestone since they could not find an off-the-shelf solution that met all of its requirements.
-
Cloudflare Previews Globally Distributed Queues without Egress Fees
Cloudflare recently announced the private beta of Cloudflare Queues, a message queuing service that allows applications to send and receive messages using Cloudflare Workers. The new service provides at-least once message delivery, supports batching of messages, and does not charge bandwidth egress fees.
-
Amazon SNS Supports FIFO for Pub/Sub Messaging
AWS has recently introduced support for First-in-First-out (FIFO) topics in Amazon SNS with strict ordering and deduplication of messages. The new feature enforces ordering in which messages are sent and received, and avoids that a message is processed multiple times.
-
Microsoft Expands the Availability of Azure Service Bus and Event Hubs
In a recent blog post, Microsoft announced Availability Zones support for Azure Service Bus Premium and Azure Event Hubs Standard. With the support, customers will have a high availability option for these services in availability zones supported regions.
-
Amazon Updates SQS to Support Amazon VPC Endpoints Using AWS PrivateLink
Amazon announced last month that its fully managed message queuing service Simple Queue Service (SQS) supports Virtual Private Cloud (VPC) Endpoints using AWS PrivateLink. Now customers can have with this update private access to SQS and not have to use public IPs and traverse the public internet.
-
Google Announces Task Queue Service Cloud Tasks
Google has announced Cloud Tasks, a task queue service for Google Cloud Platform’s App Engine service. Cloud Tasks enables asynchronous execution of tasks from applications, decoupling services and supporting the implementation of long-running and background activities.
-
Monitoring Distributed Task Queues at MeilleursAgents
MeilleursAgents, a website that lets property sellers list and get an estimated price of their property, shared details of how their Celery-based distributed task queue is monitored. A combination of Python, StatsD, Bucky, Graphite and Grafana form the pipeline to monitor task lifecycle and execution rates.
-
Chaperone - A Kafka Auditing Tool from the Uber Engineering Team
The Uber Engineering team released their Kafka auditing tool called Chaperone as an open-source project. Chaperone allows for auditing and detection of data loss, latency, and duplication of messages in the multi-datacenter and high-volume Kafka setup at Uber.
-
Amazon Simple Queue Service (SQS) Gains FIFO Queues
Amazon's Simple Queue Service (SQS) recently gained FIFO (first-in, first-out) queues, which are designed to "guarantee that messages are processed exactly once, in the order that they are sent, and without duplicates". AWS rolled out this new queue type in the US East (Ohio) and US West (Oregon) regions and "plans to make it available in many others in early 2017".