BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Amazon Simple Queue Service (SQS) Gains FIFO Queues

Amazon Simple Queue Service (SQS) Gains FIFO Queues

Amazon's Simple Queue Service (SQS) recently gained FIFO (first-in, first-out) queues, which are designed to "guarantee that messages are processed exactly once, in the order that they are sent, and without duplicates". AWS rolled out this new queue type in the US East (Ohio) and US West (Oregon) regions and "plans to make it available in many others in early 2017".

Amazon SQS is described as a "fast, reliable, scalable, fully managed message queuing service [designed to] decouple the components of a cloud application [and] transmit any volume of data, without losing messages or requiring other services to be always available".

Accordingly, a notable characteristic of the service has been the de facto absence of scaling limits for the now called 'standard' queues, insofar users can create an unlimited number of queues, and each queue provides "nearly unlimited throughput" by default. However, this effortless scaling mode requires a message processing architecture that can only guarantee best-effort ordering and at-least-once delivery.

These standard SQS queues are now complemented by a new FIFO queue type, which trades unlimited scaling for stronger message ordering and processing guarantees, based on AWS' observation that "customers have told us that FIFO applications are generally lower-throughput – 10 messages per second or lower":

  • First-In-First-Out delivery: The order in which messages are sent and received is strictly preserved.
  • Exactly-Once processing: A message is delivered once and remains available until a consumer processes and deletes it. Duplicates are not introduced into the queue within the five minute deduplication interval.
  • Limited throughput: 300 transactions per second (TPS).

The underlying architectural tradeoffs and resulting constraints are described in more detail in a technical blog post on how the Amazon SQS FIFO API works. The authors emphasize, that while "many apps perform well with SQS’s traditional super-scalable at-least-once message processing", there are some use cases that require ordering or exactly-once processing, such as "a stream of price updates" or "commands in an interactive shell session".

The default settings for FIFO queues should be appropriate to achieve strict ordering and exactly-once processing for most use cases "if your network connections don’t drop for minutes at a time, and your messages have unique identifiers". For more granular requirements, it is possible to facilitate message groups (each message that belongs to the same group is processed in order) and a dedicated deduplication ID (which otherwise defaults to the SHA256 hash of the message content).

Microsoft Azure's Service Bus also optionally guarantees FIFO message ordering and exactly-once delivery. In contrast, Google Cloud Platform’s Cloud Pub/Sub service neither supports FIFO message ordering nor exactly-once delivery, since Google argues that "there are very few use cases that actually require strict ordering".

The Amazon SQS documentation features a developer guide, including a dedicated section on FIFO queues that also provides a checklist for moving from a standard queue, as well as the API reference. A separate FAQ is also available. SQS is priced per API request (rather than per message: one request can contain up to ten messages) and the new FIFO queues are priced similarly to how standard queues have been priced before, while the pricing for standard queues has been reduced by twenty percent. Support is provided via the Amazon SQS forum.

Rate this Article

Adoption
Style

BT