Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Enhancing Observability: Amazon CloudWatch Logs Introduces Account-Level Subscription Filter

Enhancing Observability: Amazon CloudWatch Logs Introduces Account-Level Subscription Filter

This item in japanese

The recent update to Amazon CloudWatch Logs introduces support for account-level subscription filtering. With this enhancement, developers can now access a real-time feed of CloudWatch Logs from all logs groups and have it delivered to a single destination for further processing.

The implementation of a single account-level subscription filter enables customers to deliver real-time log events that are ingested into Amazon CloudWatch Logs to a Kinesis Data Stream, Amazon Kinesis Data Firehose, or AWS Lambda for custom processing, analysis, or redirection to alternative destinations. It is possible to set an account-level subscription policy that includes a subset of log groups in the account.

All the logs that are sent to a receiving service through an account-level subscription policy are base64 encoded and compressed as gzip files. Designed to reduce the overhead of managing large and complex AWS deployments, the account-level subscription filter applies to both existing log groups and any future log groups that match the configuration. Jeremy Daly, CEO and founder of Ampt, comments:

This is a dream come true for those of us who wrestle with tens (if not hundreds) of log group subscription filters.

The put-account-policy API can be used to set the CloudWatch Logs account-level subscription. For example, using the AWS CLI, the following command sends all log data to the helloWorld Lambda function, excluding the group names LogGroupToExclude1 and LogGroupToExclude2.

aws logs put-account-policy \
	--policy-name "ExamplePolicyLambda" \
	--policy-document '{"DestinationArn": "arn:aws:lambda:region:123456789012:function:helloWorld", "FilterPattern": "Test", "Distribution": "Random"}' \
	--selection-criteria 'LogGroupName NOT IN ["LogGroupToExclude1", "LogGroupToExclude2"]' \
	--scope "ALL"

Source: AWS documentation.

Regardless of the chosen destination, AWS stresses the importance of evaluating upfront the volume of log data that will be generated to avoid throttling. Developers should make sure that the Kinesis Data Firehose stream or Lambda function can handle the volume, or that the Kinesis Data Streams has enough shards. With Kinesis Data Streams, throttled deliverables are retried for up to 24 hours and then dropped.

The cloud provider warns as well about the risk of infinite recursive loops with subscription filters, triggering large increases in ingestion billing. The team provides advice on recursion prevention:

To mitigate this risk, we recommend that you use selection criteria in your account-level subscription filters to exclude log groups that ingest log data from resources that are part of the subscription delivery workflow (...) When excluding log groups, consider the following AWS services that produce logs and may be a part of your subscription delivery workflows: Amazon EC2 with Fargate, Lambda, AWS Step Functions, and Amazon VPC flow logs that are enabled for CloudWatch Logs.

Referring to the new features added at re:Invent, and the recent announcement that CloudWatch alarms now support AWS Lambda functions as an action for state changes, Ran Isenberg, principal software architect at CyberArk, notes:

CloudWatch is on a roll lately and is making up lost ground in comparison to third-party observability tools.

CloudWatch Logs Account-level Subscription Filter is available in all AWS commercial regions except Israel and Canada West.

Each AWS account can create one account-level subscription filter.

About the Author

Rate this Article