BT

Amazon Simple Queue Service (SQS) Gains FIFO Queues

| by Steffen Opel Follow 4 Followers on Dec 31, 2016. Estimated reading time: 2 minutes |

Amazon's Simple Queue Service (SQS) recently gained FIFO (first-in, first-out) queues, which are designed to "guarantee that messages are processed exactly once, in the order that they are sent, and without duplicates". AWS rolled out this new queue type in the US East (Ohio) and US West (Oregon) regions and "plans to make it available in many others in early 2017".

Amazon SQS is described as a "fast, reliable, scalable, fully managed message queuing service [designed to] decouple the components of a cloud application [and] transmit any volume of data, without losing messages or requiring other services to be always available".

Accordingly, a notable characteristic of the service has been the de facto absence of scaling limits for the now called 'standard' queues, insofar users can create an unlimited number of queues, and each queue provides "nearly unlimited throughput" by default. However, this effortless scaling mode requires a message processing architecture that can only guarantee best-effort ordering and at-least-once delivery.

These standard SQS queues are now complemented by a new FIFO queue type, which trades unlimited scaling for stronger message ordering and processing guarantees, based on AWS' observation that "customers have told us that FIFO applications are generally lower-throughput – 10 messages per second or lower":

  • First-In-First-Out delivery: The order in which messages are sent and received is strictly preserved.
  • Exactly-Once processing: A message is delivered once and remains available until a consumer processes and deletes it. Duplicates are not introduced into the queue within the five minute deduplication interval.
  • Limited throughput: 300 transactions per second (TPS).

The underlying architectural tradeoffs and resulting constraints are described in more detail in a technical blog post on how the Amazon SQS FIFO API works. The authors emphasize, that while "many apps perform well with SQS’s traditional super-scalable at-least-once message processing", there are some use cases that require ordering or exactly-once processing, such as "a stream of price updates" or "commands in an interactive shell session".

The default settings for FIFO queues should be appropriate to achieve strict ordering and exactly-once processing for most use cases "if your network connections don’t drop for minutes at a time, and your messages have unique identifiers". For more granular requirements, it is possible to facilitate message groups (each message that belongs to the same group is processed in order) and a dedicated deduplication ID (which otherwise defaults to the SHA256 hash of the message content).

Microsoft Azure's Service Bus also optionally guarantees FIFO message ordering and exactly-once delivery. In contrast, Google Cloud Platform’s Cloud Pub/Sub service neither supports FIFO message ordering nor exactly-once delivery, since Google argues that "there are very few use cases that actually require strict ordering".

The Amazon SQS documentation features a developer guide, including a dedicated section on FIFO queues that also provides a checklist for moving from a standard queue, as well as the API reference. A separate FAQ is also available. SQS is priced per API request (rather than per message: one request can contain up to ten messages) and the new FIFO queues are priced similarly to how standard queues have been priced before, while the pricing for standard queues has been reduced by twenty percent. Support is provided via the Amazon SQS forum.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss
BT