BT

AWS Releases CloudTrail Processing Library

| by Steffen Opel Follow 3 Followers on Nov 15, 2014. Estimated reading time: 2 minutes |

Amazon Web Services (AWS) recently released the AWS CloudTrail Processing Library (CPL), a "Java client library that makes it easy to build an application that reads and processes CloudTrail log files in a fault tolerant and highly scalable manner".

AWS CloudTrail records all API calls made in an AWS account for logging and auditing use cases including security analysis, change tracking, compliance aid and operational troubleshooting, as explained in more detail in our previous coverage. It has been introduced at re:Invent 2013 and expanded over the course of 2014 to support all public AWS regions and most services.

As usual, AWS provides an API for integrating CloudTrail with custom monitoring solutions. However, implementing the logic for processing CloudTrail events required interaction with at least three involved services Amazon S3, Amazon SNS and CloudTrail itself, while considering resiliency and fault tolerance – a cumbersome task.

This has now been addressed by a "new extension to the AWS SDK for Java":

The AWS CloudTrail Processing Library, or CPL, eliminates the need to write code that polls Amazon SQS queues, reads and parses queue messages, downloads CloudTrail log files, and parses and serializes events […]. Developers can read and process CloudTrail log files in as few as 10 lines of code. CPL handles transient and enduring failures […] in a resilient and fault tolerant manner. CPL is built to scale easily and can process an unlimited number of log files in parallel.

A minimum integration just requires the implementation of an EventProcessor that receives the CloudTrail log data:

public class SampleEventsProcessor implements EventsProcessor {
    public void process(List<CloudTrailEvent> events) {
        int i = 0;
        for (CloudTrailEvent event : events) {
            System.out.println(String.format("Process event %d : %s", i++, 
                event.getEventData()));
        }
    }
}

As illustrated by Jason Fulghum in his introductory post on the Java SDK blog, events can be filtered directly within this loop. More advanced use cases can be implemented by means of a few dedicated interfaces instead:

  • EventFilter – provides a callback to determine whether or not to process a log record
  • EventProcessor – provides a callback to deliver log records for processing (see above)
  • ExceptionHandler – provides a callback that handles exceptions while processing log files
  • ProgressReporter – provides an interface for custom handling of progress
  • SourceFilter – provides a callback to determine whether or not to process a log source

An EventFilter allows to selectively process events based on the granular information within the CloudTrailEventData, for example the AWS region, the originating service or even IP address, the user identity, and of course an event name such as "DeleteSecurityGroup". The CPL provides sample implementations for all interfaces, including an EventFilter that only surfaces EC2 deletion API calls:

public class SampleEventFilter implements EventFilter{
    private static final String EC2_EVENTS = "ec2.amazonaws.com";
 
    @Override
    public boolean filterEvent(CloudTrailEvent event) throws CallbackException {
        CloudTrailEventData eventData = event.getEventData();
 
        String eventSource = eventData.getEventSource();
        String eventName = eventData.getEventName();
 
        return eventSource.equals(EC2_EVENTS) && eventName.startsWith("Delete");
    }
}

The AWS CloudTrail Processing Library source code is available on GitHub. The CloudTrail documentation features a section on how to use CPL as well as the CPL API reference. Support is offered via issue tracker and the AWS CloudTrail forum.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT