Recently, Amazon announced a new log aggregation service called AWS Firelens. The service unifies log filtering and routing across all AWS container services including Amazon ECS, Amazon EKS, and AWS Fargate.
Martin Beeby, a principal evangelist for Amazon Web Services, wrote in his blog post about the usage of the log aggregation service Firelens:
Using FireLens, customers can direct container logs to storage and analytics tools without modifying deployment scripts, manually installing extra software or writing additional code. With a few configuration updates on Amazon ECS or AWS Fargate, you select the destination and optionally define filters to instruct FireLens to send container logs to where they are needed.
Two popular open source logging projects Fluentd and Fluent Bit work seamlessly with Firelens. Moreover, users can leverage either one to directly stream logs to Amazon CloudWatch, Amazon Kinesis Data Firehose, or partner products like Datadog, Splunk, and New Relic. Amazon provides users with an AWS for Fluent Bit image or the ability to use their own Fluentd or Fluent Bit image.
Source: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/
To enable routing to a destination, users can create and configure a task definition using either AWS SDKs, AWS CLI, or AWS Management Console. Furthermore, users need to create an Identity Access Management (IAM) role for the tasks to provide the necessary permissions for any AWS services that the tasks require. For example, if a container is routing logs to Kinesis Data Firehose, then the task would need permission to call the firehose:PutRecordBatch API.
One of AWS partners Sysdig has integrated Falco, an open-source Kubernetes runtime security project currently donated to CNCF, with Fluent Bit. The integration can enable AWS users to stream Falco security data into AWS FireLens for easier log management. In a press release of Sysdig about the Sysdig integration with Falco, Kris Nova, chief open source advocate at Sysdig, said:
AWS asked Sysdig to join the FireLens preview program because AWS values Falco’s ability to secure cloud-native environments. By integrating with FireLens, we hope to make it easier for all organizations to develop in the cloud, secure in the cloud, audit in the cloud, no matter their approach.
Beside AWS with Firelens, other cloud providers provide log aggregating capabilities such as the Microsoft with Azure Log Analytics feature in Azure Monitor. This Azure feature enables its users to also receive Kubernetes logs from Fluent Bit in case clusters are not running in Azure Container Services (AKS) through the supported output plugins.
Lastly, AWS Firelens is currently available in all regions that support Amazon ECS, and AWS Fargate. Furthermore, sample logging architectures for FireLens on Amazon ECS and AWS Fargate are available on GitHub, and other details are available on the AWS documentation page.