AWS has outlined a specification-driven composition pattern for building flexible data transformation workflows, separating workflow intent from processing logic to reduce duplicated pipeline code and simplify validation and governance. The approach uses structured specifications to dynamically assemble reusable processing capabilities, with a serverless implementation based on AWS Lambda, AWS Step Functions, Amazon S3, and Amazon OpenSearch Service.
The pattern targets data pipelines that become difficult to maintain as organizations add datasets and workflow variants. AWS notes that script-based implementations often combine orchestration, transformation, and validation logic, necessitating code changes and redeployments when new datasets are introduced. The resulting duplication can also make workflow behavior harder to trace, particularly in regulated environments.
Specification-driven composition separates the workflow into three layers: an intent layer containing the specification, a composition layer that validates the specification and assembles the pipeline, and a processing layer that executes the transformation steps.
Three-layer specification-driven composition architecture (Source: AWS Blog Post)
Rajat Shrivastav, commenting on AWS's description of the pattern on LinkedIn, wrote,
By codifying the intent into a declarative specification, you ensure that traceability isn't an afterthought; it's the core of the pipeline.
A specification, typically represented as JSON or YAML, describes source and target datasets, field mappings, and transformations without defining how those transformations are implemented. A composer validates the specification, checks referenced capabilities, and generates a runnable workflow. A capability registry maintains metadata for reusable transformation functions, including identifiers, input and output formats, invocation details, permissions, and versions.
The AWS example stores workflow specifications in Amazon S3. An S3 event invokes a Lambda-based composer, which validates the specification and queries Amazon OpenSearch Service for capability metadata. The composer then creates and starts an AWS Step Functions state machine that invokes individual Lambda-based capability processors. The processors emit traces to Amazon CloudWatch Logs.
AWS implementation of Specification-Driven Composition(Source: AWS Blog Post)
The registry also enables capability discovery using metadata such as descriptions, schemas, and tags. Specifications can reference reusable transformations by intent rather than requiring authors to know an exact implementation identifier. AWS also describes versioned capability references to support reproducible workflow execution and separate validation and approval of new capabilities.
The pattern can incorporate data classification into the specification. Sensitive fields can be tagged, while capabilities declare how they affect sensitivity. The composer can use those definitions to validate the resulting classification and generate masking artifacts for downstream consumers.
AWS positions the pattern for regulated reporting, multi-source integration, and reusable ETL workflows where traceability and pre execution validation are important. The company also notes that specification-driven composition can introduce unnecessary complexity for simple transformations or environments with only a few workflows, making the approach more applicable as workflow variation and governance requirements increase.