BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News AWS Step Functions Gains Integrations with over 200 Additional Services

AWS Step Functions Gains Integrations with over 200 Additional Services

This item in japanese

Bookmarks

Amazon Web Services (AWS) recently announced that AWS Step Functions supports over 200 AWS services via an AWS SDK integration, thereby expanding the number of supported API actions from 46 to over 9,000. The new integration type can support most existing and future services and API actions going forward.

AWS Step Functions is a "low-code visual workflow service" used to orchestrate AWS services, automate business processes, and build serverless applications. The service offers two distinct workflow types for use cases that target either auditable and potentially long-running workflows (up to a year) or high-volume event processing workloads (up to five minutes). Typical workflow orchestration aspects like "failures, retries, parallelization, service integrations, and observability" are fully-managed so that "developers can focus on higher-value business logic" (previous coverage).

So far, Step Functions offered now so-called optimized integrations with 17 AWS services. Former AWS distinguished engineer Tim Bray, who "did a whole lot of work" on the service's creation, explains that "optimized" means "it's smart about the way it calls the service and can operate in either fire-and-forget or wait-for-completion mode [and] can autogenerate IAM policies to make your life easier". This more ambitious developer experience resulted in the list of supported AWS services growing comparatively slowly, with the ability to send custom events to Amazon EventBridge being the latest integration published in May 2021.

To remove the added cost and complexity for working around unavailable integrations with custom code invoked via the ubiquitous AWS Lambda integration, Step Functions now provides a generic AWS SDK integration that can support most services and API actions going forward, with exceptions "for a variety of reasons". This flexibility comes at the expense of having to manually configure the otherwise autogenerated IAM policies. The Amazon States Language (ASL) syntax for the new feature follows the well-established ARN based format for URI-based integration:

arn:aws:states:::aws-sdk:serviceName:apiAction.[serviceIntegrationPattern]

In a popular tweet, Bray summarized the expected impact of this "no-code invocation, from Step Functions, of any API defined in the AWS SDK":

Hear that screaming? That’s the sound of a huge number of utility-glue Lambda functions realizing they are now superfluous.

In the launch blog post, AWS senior developer advocate Marcia Villalba walks through a comprehensive workflow example that uses several SDK integrations to fetch a media file from an S3 bucket, then transcribes the video, and finally translates the transcript into the target language.

AWS Step Functions SDK integration example

Image: AWS Step Functions SDK integration example (via launch blog post)

Notably, the transcription job is an asynchronous operation that requires a polling loop before moving to the next workflow state. While polling loops are straight forward to implement, AWS Serverless Hero Ben Kehoe cautioned that such use cases would still benefit from optimized integrations:

Right, but while the AWS SDK integration allows for "you can do it today", it should not be an excuse for Step Functions not to build sync integrations! Polling loops in state machines are serious undifferentiated heavy lifting.

On a related note, AWS Community Hero Ben Bridts wishes "pagination support for Step Functions SDK integrations" to deal with larger API responses:

Building pagination in the Amazon States Language is giving me a headache, and the SDK already knows how to do it [...] Bonus-points for also supporting SDK-based waiters.

Some of these limitations can be worked around by higher level tooling. For example, the Cloud Development Kit (CDK) support for the AWS SDK integration pattern assists with IAM policy generation by automatically adding an IAM statement to the state machine role's policy "based on the service and action called" and only requires intervention in the few cases "where the IAM action name does not match with the API service/action name":

declare const myBucket: s3.Bucket;
const getObject = new tasks.CallAwsService(this, 'GetObject', {
  service: 's3',
  action: 'getObject',
  parameters: {
    Bucket: myBucket.bucketName,
    Key: sfn.JsonPath.stringAt('$.key')
  },
  iamResources: [myBucket.arnForObjects('*')],
});

In related news, AWS Athena and AWS Batch have integrated AWS Step Functions into their service consoles to visualize workflows with contextual deep links to queries and jobs, and to ease getting started via sample projects for common use cases.

Microsoft's Azure Logic Apps service uses a different extensibility architecture based on connectors and provides many built-in connectors for other Azure services, plus the ability to build a custom connector for more tailored scenarios. Google Cloud Platform's Workflows also publishes dedicated connectors to access other Google Cloud products.

The AWS Step Functions documentation features a developer guide, including a section on using Step Functions with other services, and the API reference. The AWS CLI Step Functions commands are documented separately. Support is provided via the AWS Step Functions forum. AWS SDK integrations are provided at no additional charge beyond the regular usage-based pricing for Step Functions and each target service.

Rate this Article

Adoption
Style

BT