Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Amazon SNS Gains Message Archiving and Analytics via Amazon Kinesis Data Firehose

Amazon SNS Gains Message Archiving and Analytics via Amazon Kinesis Data Firehose

This item in japanese

Amazon Web Services (AWS) recently announced that Amazon SNS supports Amazon Kinesis Data Firehose subscriptions to send messages to "data lakes, data stores, and analytics services [...] without writing custom code". The new event destination also simplifies the integration of third-party service providers.

Amazon Simple Notification Service (Amazon SNS) is a managed service that provides asynchronous message delivery from publishers to subscribers "for both application-to-application (A2A) and application-to-person (A2P) communication". It supports a wide variety of native AWS event sources to implement high throughput and low latency event-driven architectures (previous coverage).

Amazon Kinesis Data Firehose is a managed service to "prepare and load real-time data streams into data stores and analytics services" without the need to implement anything but an optional serverless data transformation via an AWS Lambda function (previous coverage).

Amazon SNS has previously supported general-purpose A2A event destinations like Amazon SQS, AWS Lambda, and SNS aware HTTP/S endpoints, which required custom solutions for popular destinations with more specific requirements. AWS has now added an Amazon Kinesis Data Firehose event destination that can in turn send records to delivery stream destinations including Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and "any HTTP endpoint owned by you or any of your third-party service providers". Coupled with the built-in Kinesis Data Firehose data transformation capability, this obsoletes most custom code to "capture, transform, buffer, compress and upload data" en route to these destinations.

Amazon SNS event destinations

Image: Amazon SNS event destinations (via AWS documentation)

The introductory blog post outlines several example use cases that are now easier to achieve:

The new Kinesis Data Firehose integration supports regular SNS features such as dead-letter queues (DLQs) and message filtering, which allows to "offload the message filtering logic from subscribers and the message routing logic from publishers".

It is worth noting that the event bus based Amazon EventBridge service also provides DLQscontent filtering, input transformation, and has native event archive and replay capabilities (previous coverage). James Beswick, principal developer advocate for AWS Serverless, acknowledges in a blog post on choosing between messaging services that "SNS and EventBridge have many similarities":

Both can be used to decouple publishers and subscribers, filter messages or events, and provide fan-in or fan-out capabilities. However, there are differences in the list of targets and features for each service, and your choice of service depends on the needs of your use-case.

The post then describes the EventBridge exclusive SaaS partner event source and schema registry capabilities and provides a detailed feature comparison table. Beswick also followed up with example use cases combining features of both services in a subsequent blog post on building resilient serverless patterns.

In related news, Amazon SNS recently introduced additional operators for its message filtering capability, meanwhile supports ten-digit long codes and toll-free numbers for sending SMS to US destinations, and also improved Amazon CloudWatch monitoring by publishing metrics at 1-minute resolution.

The Amazon SNS documentation features a developer guide, including a section on message archiving and analytics, the AWS CLI reference, and the API reference. Amazon SNS is subject to usage-based pricing that is primarily based on notification volume and varies by destination. Charges associated with destination services like Amazon SQS, AWS Lambda, and Amazon Kinesis Data Firehose are billed separately. Support is provided via the Amazon SNS forum.

Rate this Article