A Comprehensive Guide to Building Event-Driven Architecture on Azure, AWS, and Google Cloud

In this article, you’ll find guidance to Azure, AWS, and Google Cloud resources, along with unique architecture examples that incorporate the AWS EventBridge, SNS, Azure Service Bus, Eventgrid, and Google Cloud Eventarc. These examples can help you better grasp the resources’ concepts and enable you to kickstart building your own architecture using an event-driven approach.

AWS SQS, Azure Service Bus, and Google Pub/Sub

The AWS Simple Queue Service (SQS) is a message queue system that operates on a first-in, first-out (FIFO) basis, allowing easy sending, storage, and message retrieval. This service is suitable for simple to intermediate scenarios, such as connecting multiple services to prevent data loss and direct connections. SQS offers several features, including at-least-once delivery, which ensures that messages are delivered, and standard FIFO, which delivers messages in a random order using the traditional first-in, first-out method.

However, for complex architectures that depend on multiple services, SQS may not be the optimal choice due to its message size and retention period restrictions, which can impact the scalability and performance of the system.

Google Cloud has Google Cloud Pub/Sub, a messaging service in the Google Cloud Platform. It provides asynchronous messaging between independent components or services in a scalable and reliable manner. Pub/Sub offers features like at-least-once delivery, push and pull message delivery modes, and a topic-based publish-subscribe model.

ASO Azure has managed a service called Az Queue. It is designed to handle large numbers of messages efficiently and reliably. Messages can be up to 64 KB in size, and there is no limit to the number of messages you can store in a queue. Messages are stored in a first-in-first-out (FIFO) order, which ensures that messages are processed in the order they are received.

For complex scenarios, Azure can offer an Azure service bus. Azure Service Bus is a fully managed messaging service allowing reliable message delivery between applications or services. It supports pub/sub and queue-based messaging patterns and provides features such as message ordering, dead-lettering, and massage sessions. It also offers advanced features such as auto-forwarding and partitioning for high scalability.

Alongside Queues, Azure Service bus provides Topics. Topics support publisher-subscriber messaging patterns with multiple subscribers receiving the same message.

Let’s look at the architecture example of event-driven and microservice architecture to find the plagiary content. Similar systems have been integrated into higher education institutions and patent offices.

Figure 1. Detecting Plagiarism Solution Architecture

[Click on the image to view full-size]

Azure Service Bus is a central service that facilitates the exchange of processed messages among various services represented by Azure functions. Each function represents a single piece of functionality. Functions like Google API, RSS, and blogs serve as proxies between API sources, and their purpose is to connect, submit, and process data. On the other hand, services such as Object Detection and Text Translation represent AI content parsing and are connected to Azure Cognitive and Computer Vision Services.

Let’s look into a Terraform script example of a highly available Azure Service Bus with queue.

resource "azurerm_resource_group" "rg_service_bus" {
  name     = "example-resource-group"
  location = "westus"
}

resource "azurerm_servicebus_namespace" "ns-service-bus" {
  name                = "example-service-bus-namespace"
  location            = azurerm_resource_group.rg_service_bus.location
  resource_group_name = azurerm_resource_group.example.name

  sku {
    name     = "Standard"
    tier     = "Standard"
    capacity = 1
  }

  tags = {
    environment = "production"
  }
}

resource "azurerm_servicebus_queue" "service-bus-queue" {
  name                = "example-queue"
  namespace_name      = azurerm_servicebus_namespace.ns_service_bus.name
  resource_group_name = azurerm_resource_group.rg_service_bus.name

  enable_partitioning = true
  message_count       = 0

  tags = {
    environment = "production"
  }
}

resource "azurerm_servicebus_namespace_authorization_rule" "example" {
  name                = "example-auth-rule"
  namespace_name      = azurerm_servicebus_namespace.ns_service_bus.name
  resource_group_name = azurerm_resource_group.example.name

  listen = true
  send   = true
}

output "service_bus_connection_string" {
  value = azurerm_servicebus_namespace.ns_service_bus.default_primary_connection_string
}

This script creates a resource group, a Service Bus namespace with a Standard pricing tier, a queue within the namespace with partitioning enabled, and an authorization rule with listen and send permissions. It also outputs the connection string for the Service Bus namespace.

AWS SNS, EventBridge, Azure Event Grid, and Google EventArc

Here, we start with a service that provides simple notification logic, which is Simple Notification Service (SNS). The SNS is a managed notification service, with the primary feature being a topic-based approach, which allows it to disseminate messages from a single source application to multiple subscribers. The following protocols are supported by SNS, enabling a range of subscribers:

• SQS
• HTTP / HTTPS
• Email
• Email-JSON
• SMS
• Lambda
• Firehose

The main disadvantage of AWS SNS is that it may not provide as much control over message delivery compared to other messaging services, such as AWS SQS. SNS does not guarantee the order of message delivery and may deliver messages more than once in some cases, which can be problematic for certain use cases.

SNS is quite suitable for building simple notification architecture. An excellent example of building budget notifications and assigning events when the budget exceeds limits.

In this architecture, the producer sends a message to SNS, and SNS delivers that message to all the interested consumers, such as email subscribers, mobile devices, or HTTP/S endpoints.

Figure 2. AWS Budget Notification Architecture

[Click on the image to view full-size]

The architecture includes the following components:

1. SNS topic: In the AWS Management Console, create an SNS topic with a name that indicates it is for budget notifications. The example code below sets up an SNS topic called "Budget_Alerts" and subscribes to an email address to it. It also creates a budget for your AWS account with a limit of $100 for the "AWS Lambda" and "Amazon EC2" services. Example:
    

   import boto3
   
   # Create a client for the SNS service
   sns = boto3.client('sns')
   
   # Set up a topic for budget alerts
   response = sns.create_topic(
       Name='Budget_Alerts'
   )
   topic_arn = response['TopicArn']
   
   # Subscribe an email to the topic to receive notifications
   response = sns.subscribe(
       TopicArn=topic_arn,
       Protocol='email',
       Endpoint='your-email@example.com'
   )
   
   # Set up a budget for your AWS account
   client = boto3.client('budgets')
   response = client.create_budget(
       AccountId='your-aws-account-id',
       Budget={
           'BudgetName': 'MyBudget',
           'BudgetLimit': {
               'Amount': '100.0',
               'Unit': 'USD'
           },
           'CostFilters': {
               'Service': ['AWS Lambda', 'Amazon EC2']
           },
           'BudgetType': 'COST',
           'Notification': {
               'NotificationType': 'ACTUAL',
               'Threshold': 90.0,
               'ThresholdType': 'PERCENTAGE',
               'ComparisonOperator': 'GREATER_THAN',
               'SubscriberEmailAddresses': [
                   'your-email@example.com'
               ]
           },
           'TimeUnit': 'MONTHLY',
           'TimePeriod': {
               'Start': '2022-01-01',
               'End': '2022-12-31'
           }
       }
   )
   

2. We can add the email addresses or phone numbers of the people who need to receive budget notifications to the SNS topic. We can also choose the desired protocol for each subscriber.
3. In the AWS Budgets console, we can create a budget that monitors your monthly spending and sends an alert when your spending exceeds a certain threshold.
4. Configures SNS as the notification channel: In the budget alert settings, we select SNS as the notification channel and specify the ARN (Amazon Resource Name) of the SNS topic you created in step 1.
5. Trigger a budget alert by exceeding the spending threshold, and verify that the subscribers receive the notification via their chosen protocol.

AWS also has a service that provides way more options to build event-driven architecture - AWS EventBridge. It allows us to develop scalable event-driven applications easily. It provides the ability to create rules that match events from various AWS services and custom applications, which can then be directed to one or multiple targets, including other AWS services, AWS Lambda functions, and SNS topics. EventBridge allows us to create custom event buses that can be used to isolate events and manage access to them. EventBridge can be used to build a wide variety of event-driven applications, including serverless applications, microservices, and event-driven data processing pipelines.

Let’s have a look at an example of EventBridge. Below is the smart lighting system architecture that uses AWS Event Bridge as a serverless event bus.

Figure 3. AWS Smart Lighting Architecture

[Click on the image to view full-size]

The architecture includes the following components:

1. IoT Light Devices: Smart light bulbs that connect to the internet via IoT. These devices receive and process instructions to turn on/off and change color and brightness.
2. AWS IoT Core / Greengrass: A managed cloud service that enables secure communication and management of IoT devices.
3. AWS IoT Events: A service that detects and responds to events from IoT devices and applications.
4. AWS EventBridge: A serverless event bus that simplifies the integration of event-driven architectures.
5. Lambda Functions containing business logic to process the device’s state and settings and save them to the database. And function to integrate messages with Slack bot.

Below is the code example that deploys an EventBridge rule for a smart lighting system that triggers a Lambda data processor.

import boto3
import os

iot_client = boto3.client('iot')
eventbridge_client = boto3.client('events')
lambda_client = boto3.client('lambda')

# Create an AWS IoT Core rule to forward messages to EventBridge
response = iot_client.create_topic_rule(
    ruleName='smart-lighting-rule',
    topicRulePayload={
        'sql': "SELECT * FROM 'smart-lighting-topic'",
        'actions': [
            {
                'eventBusName': os.environ['EVENT_BUS_NAME'],
                'lambda': {
                    'functionArn': os.environ['MESSAGE_PROCESSOR_ARN']
                }
            }
        ]
    }
)

# Create an EventBridge rule to trigger the Lambda function
response = eventbridge_client.put_rule(
    Name='smart-lighting-event-rule',
    EventPattern={
        'detail-type': ['aws.iot.receive'],
        'source': ['aws.iot'],
        'detail': {
            'topic': ['smart-lighting-topic']
        }
    },
    State='ENABLED'
)

# Add a target to the EventBridge rule to trigger the Lambda function
response = eventbridge_client.put_targets(
    Rule='smart-lighting-event-rule',
    Targets=[
        {
            'Id': '1',
            'Arn': os.environ['LAMBDA_FUNCTION_ARN']
        }
    ]
)

# Grant permissions to the Lambda function to receive messages from IoT Core
response = lambda_client.add_permission(
    FunctionName=os.environ['LAMBDA_FUNCTION_NAME'],
    StatementId='IoTReceiveMessage',
    Action='lambda:InvokeFunction',
    Principal='iot.amazonaws.com',
    SourceArn=os.environ['IOT_CORE_TOPIC_ARN']
)

This code creates an AWS IoT Core rule to forward messages to EventBridge, creates an EventBridge rule to trigger a Lambda function, and grants the Lambda function permission to receive messages from IoT Core.

Azure and Google Cloud have their own services to build event-driven architecture. Azure provides Event Grid. It is a fully managed event routing service that enables event-based communication between different services and applications within Azure and external sources. Event Grid supports various event sources, including Azure services, custom applications, and third-party services. It integrates with various Azure resources like Blob Storage, Azure Functions, Event Hubs, and more. We can see this in the following image.

Figure 4. Azure Event Grid Sources and Handlers (source)

[Click on the image to view full-size]

Google Cloud has Eventarc, a "brother" of Azure Event Grid. Eventarc is a managed event ingestion and delivery service that enables developers to trigger serverless functions or applications in response to events from Google Cloud services or third-party sources. Eventarc simplifies the configuration and management of event-driven architectures by abstracting the underlying infrastructure and providing a unified interface for event sources and destinations.

Eventarc can be used in many scenarios where we need to trigger an action based on an event within your GCP infrastructure. One is to monitor your cloud infrastructure in real-time and trigger alerts or actions when specific events occur.

For example, you might use Eventarc to monitor your Google Kubernetes Engine cluster for changes in pod status or resource utilization and trigger alerts or autoscaling actions as needed.

Figure 5. GKE Cluster Monitoring and Management Architecture

[Click on the image to view full-size]

1. Prometheus as a Monitoring Agent: We have a Prometheus agent installed on our GKE cluster that collects metrics and generates events whenever a pod status or resource utilization changes.
2. Eventarc trigger: Eventarc trigger that listens for events from the monitoring agent.
3. Cloud Function: Cloud Function triggered by the Eventarc trigger. This Cloud Function performs necessary processing or integration tasks, such as sending alerts or triggering autoscaling actions.
4. Alerting system: We use an alerting system, such as Google Cloud Monitoring, to notify relevant stakeholders when specific events occur, such as when pod status changes, or resource utilization exceeds a threshold.
5. Autoscaling system: We use an autoscaling system, such as Google Cloud Kubernetes Autoscaler, to automatically scale your GKE cluster up or down based on changes in resource utilization.

Deploy Event Arc trigger that listens for events from a Prometheus agent in GKE. We need to do the following steps:

1. Install Prometheus on your GKE cluster and configure it to scrape the metrics from your applications and services running on it.
2. Create an Eventarc trigger that listens for events from the Prometheus agent by running the following command:

gcloud beta eventarc triggers create monitoring-trigger \
--destination-func=”kube_manager”  \
--event-filters="type=google.cloud.audit.log.v1.written" \
--event-filters="serviceName=pubsub.googleapis.com" \
--event-filters="methodName=google.pubsub.v1.Publisher.Publish" \
--event-filters="resource.type=pubsub_topic" \
--event-filters="resource.name=projects/<project-id>/topics/<topic-name>" \
--attribute-filter="metadata.name=metric_name" \
--attribute-filter="resource.labels.cluster_name=gke-cluster-name" \
--service-account=<service-account-email> \
--retry

It is also worth mentioning that AWS has their version of managed service to build observability solutions. It’s called AWS CloudWatch, and it supports monitoring various AWS services such as EC2, RDS, S3, Lambda, and many others. It can also monitor custom metrics and logs. CloudWatch allows you to set up alarms and notifications.

AWS Kinesis, Azure Event Hub, and Amazon MSK

AWS Kinesis and Azure Event Hub are services designed to handle real-time streaming data. They offer the ability to collect, stream, and analyze data in real-time.

AWS Kinesis offers several features. One of them is the Firehose managed service, which captures and stores real-time data for up to 60 seconds. It also supports various data sources like Splunk, S3, RedShift, and Elasticsearch. Additionally, Kinesis Stream and Analytics can be used to develop video and audio streaming solutions. For instance, one can stream data from cameras or mobile devices to Kinesis and use AWS Rekognition or SageMaker to analyze the stream.

As an example, we can set up IP camera streaming architecture.

Figure 6. AWS Video Streaming with AI Object Recognition Architecture

[Click on the image to view full-size]

Below is a short example of the code to deploy an AWS Kinesis video stream.

import boto3

# Set up the Kinesis Video Streams client
client = boto3.client('kinesisvideo')

# Create a new Kinesis Video Stream
response = client.create_stream(
    StreamName='my-stream',
    MediaType='video/h264',
)

# Get the ARN of the newly created stream
stream_arn = response['StreamARN']

# Set up the Kinesis Video Streams Signaling client
signaling_client = boto3.client('kinesis-video-signaling')

# Get the endpoint for the Signaling channel
response = signaling_client.get_signaling_channel_endpoint(
    ChannelARN=stream_arn,
    SingleMasterChannelEndpointConfiguration={
        'Protocols': ['WSS', 'HTTPS'],
        'Role': 'MASTER'
    }
)

# Get the signaling channel endpoint
signaling_channel_endpoint = response['ResourceEndpointList'][0]['ResourceEndpoint']

This code creates a new Kinesis Video Stream with the name 'my-stream' and media type 'video/h264'. It then retrieves the ARN of the new stream and uses that ARN to get the signaling channel endpoint. Finally, it prints out the signaling channel endpoint.

Azure has an Event Hub service. Azure Event Hub is a data streaming platform and event ingestion service that can handle millions of events per second. It provides a highly scalable, distributed platform for ingesting, processing, and storing data from different sources like applications, devices, and IoT sensors. Event Hub can be integrated with various Azure services for data analysis, including Stream Analytics, HDInsight, and Machine Learning.

While both services provide capabilities for processing streaming data, some differences exist. Event Hub provides a dedicated platform for event ingestion. It can integrate well with other Azure services, while Kinesis provides a broader set of services for data streaming processing and is also compatible with other AWS services.

Another powerful and fully managed service from Amazon is Managed Streaming for Apache Kafka (Amazon MSK), which makes it easy to build and run applications that use Apache Kafka as a streaming data platform. Amazon MSK handles the infrastructure provisioning, configuration, scaling, and maintenance of Apache Kafka clusters, allowing you to focus on our applications. It replicates data across multiple Availability Zones (AZs) within a region, ensuring high availability and durability. It offers various security features, including encryption at rest and in transit, integration with AWS Identity and Access Management (IAM), and VPC connectivity options. Amazon MSK seamlessly integrates with AWS services like AWS Lambda, Amazon Kinesis Data Firehose, and Amazon Managed Streaming for Apache Flink (a stream processing service).

Conclusion

This article provides a detailed guide to building event-driven architecture using cloud resources from Azure, AWS, and Google Cloud. Specifically, the article focuses on cloud components used in the middle-level architecture, such as Simple Queue Service (SQS) and Simple Notification Service (SNS) from AWS and Azure Queue and Event Grid from Azure. The article also explores key differences between Azure Service Bus and AWS MQ, managed messaging services that allow reliable message delivery between applications or services. With these cloud resources, developers can quickly understand the idea of the resources and start building their architecture based on an event-driven approach.