BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles Deploying Docker Containers Using an AWS CodePipeline for DevOps

Deploying Docker Containers Using an AWS CodePipeline for DevOps

This item in japanese

Bookmarks

Key Takeaways

  • AWS CodePipeline is a DevOps service for Continuous Integration, Continuous Delivery and Continuous Deployment of applications hosted on  various AWS platforms.
  • Amazon Elastic Container Service (ECS) is an AWS managed service for containerized applications for Docker containers.
  • Amazon Fargate is a serverless launch type for Amazon Elastic Container Service (ECS).
  • For the example application deployed to ECS an AWS CodePipeline consists of a source code repository such as a GitHub repo, AWS CodeBuild for Build and a AWS ECS (Fargate) service for Staging.
  • The benefit of using an AWS CodePipeline for an AWS ECS service is that the ECS service continues to run while a new Docker image is built and deployed.

Docker containers may be deployed using one of the several cloud platforms, Amazon Elastic Container Service (ECS) being one of them. ECS provides the  Fargate launch type, which is a serverless platform with which a container service is run on Docker containers instead of EC2 instances.

Problem

A Docker container deployment may need to be updated or modified due to changes in the Docker image source code or code build. Any modifications in the source code for a Docker image would require that a Docker image be rebuilt and the Docker service be redeployed. Without a mechanism to integrate, build, deliver and deploy source code while an AWS ECS deployment is running, it would involve stopping the ECS service tasks and as a result, incurring downtime for an ECS service.  High availability of an ECS service being a priority, stopping tasks to redeploy an application is not a suitable option.

Solution

AWS CodePipeline is a DevOps service for Continuous Integration, Continuous Delivery and Continuous Deployment of applications hosted on the various AWS platforms, including Amazon ECS and Fargate as a deployment platform. An ECS service may be updated or modified without stopping the ECS service tasks. AWS CodePipeline provides high availability of an ECS service in a dynamic environment in which source code changes for a Docker image are common. A CodePipeline consists of three phases: Source code integration, Source code build, and Deployment, as shown in Figure 1.

Figure 1. CodePipeline Phases

For source code we shall use a Github repository.  For source code build we shall use a AWS CodeBuild project. For deployment we shall use an ECS service of launch type Fargate.

Creating and deploying a CodePipeline application to ECS Fargate involves the following procedure:

  1. Create an ECS Fargate Task Definition an Service
  2. Configure connectivity on Task Subnet/s
  3. Create or Configure a S3 Bucket for Output Artifacts from the CodePipeline Build Stage
  4. Create a CodePipeline to deploy a Docker platform application (image) on ECS Fargate
  5. Modify Input/Output Settings for Stage Artifacts
  6. Run the CodePipeline
  7. Make source code modifications to re-run CodePipeline

Setting the Environment

The only prerequisite is an AWS account. The application deployed by a CodePipeline on ECS Fargate is a Docker application. Any Docker image that has source code repo could be used and we have used Docker image dvohra/node-server. Here is the GitHub source code repository for the Docker image dvohra/node-server.

Creating a GitHub Code Repository

If a new GitHub source code repository were to be used, it must include a Dockerfile from which to build Docker image. The Dockerfile for the dvohra/node-server image is based on the Docker image node:4.6. Dockerfile instructions copy a server.js file, which is used to create a Node server, to the current directory, expose port 8080 for the Node server to listen on, and run a node command on the server.js script. The server.js file creates a Node server and handles an HTTP request/response.

Adding a Build Spec for CodeBuild Project

A build spec is a YAML syntax file with build commands and settings used by a CodeBuild project to run a build. The build spec file must be called “buildspec.yml” and must be copied to the root of the source code repository.  A buildspec.yml file consists of key/value pairs to describe the various phases of a build. The build phases are represented with the phases sequence, which is a required mapping in a buildspec.yml. The version is the other required mapping in a buildspec.yml. The buildspec.yml file is listed on the GitHub repo.

Adding an Image Definitions File

For deploying Container based applications such as those deployed to ECS, AWS CodePipeline requires an Image definitions file in JSON format. The Image definitions file is called imagedefinitions.json by default but could be given another name. The image definitions file describes the container application and consists of two attributes: name and imageURI. The name specifies the Docker container name and the container must be running prior to running the CodePipeline. The imageURI specifies the Docker image to be run in the Docker container. The Docker image would typically be the same as the Docker image already running in an ECS container. The image could be different and the variation would typically be of the image tag.   The imagedefinitions.json used for the Node server application deployed to ECS Fargate is listed on the GitHub.

Creating a Task Definition

A task definition in an ECS application describes the container/s in the ECS deployment. In this section, we shall create task definition for a Node Server container to be deployed on ECS Fargate. Open this URL and log in if not already logged in.. Click on Get started to access the ECS Console. Click on Task Definitions in the navigation margin. In the Task Definitions click on Create new Task Definition. In the Create new Task Definition select launch type compatibility as Fargate. Click on Next step. Next, configure task and container definitions.  In the Add container dialog specify a Container name (node-server) and specify Image as dvohra/node-server.  The task definition is shown in Figure 2.

Figure 2. Task Definition

Configuring Connectivity in Task Subnets

Before creating a service we need to configure connectivity to the Internet in the Subnets to be used when configuring the service. The Route Table lists the routes. We need to add a route with a default Internet gateway.  Add a route with Destination as 0.0.0.0/0, which is an IP address. Select Target as an internet gateway. 

Creating and Testing the Container Service

Next, create an ECS container service in the default cluster as shown in Figure 3.  

Figure 3. Cluster with 1 Service

With Fargate launch type an Elastic Network Interface (ENI) is provisioned for each task.  Copy the Public IP of the task, which is the same as the Public IP of the Elastic Network Interface, from either the task Details page Network section or the ENI Console. Open the URL <Public IP of Task>:8080 in a browser to invoke the Node Service application. The Node Server returns a message as shown in Figure 4.

Figure 4. Node Server Response

Creating or Configuring an S3 Bucket

The CodePipeline to build the Source code for the Node server application and deploy a Docker image to an ECS service requires that the CodeBuild project generate “Output Artifacts”. The Output artifacts are stored in an S3 bucket, which is selected when creating a CodePipeline.  Create a new S3 Bucket in the S3 Console, or alternatively select a S3 bucket that may be created from an earlier run of CodePipeline. 

Creating a CodeBuild Project

Next, create a CodeBuild project that is to be used to build the source code into a Docker image. The source code for a Docker image dvohra/node-server and the buildspec.yml file used to build the source code into the Docker image was discussed earlier.  After a Docker image is built, it is uploaded to Docker hub by CodeBuild. To create a CodeBuild project, open this URL in a web browser. Select Build projects and click on Create project as shown in Figure 5.

Figure 5. Build projects>Create project

The Create project wizard gets started. In Configure project specify a Project name (node-server).  In Source>Source Provider select GitHub. For Repository select Use a repository in my account and Choose a repository as the dvohra/docker-node-server  repo. Keep the default setting for Git clone depth as 1.

Select the options for Webhook, Insecure SSL and Build Badge. Selecting Webhook makes the code get rebuilt every time a code update is made in the GitHub repo. The Insecure SSL option makes the code build SSL warnings when connecting to project source. The Build Badge option makes the project’s status visible and embeddable. In Environment: How to build select Environment image as Use an image managed by AWS CodeBuild. Select Operating System as Ubuntu.  For Runtime, select Docker as shown in Figure 6. 

Figure 6. Selecting Runtime as Docker

For Runtime version, select aws:codebuild/docker:17.09.0, which represents the Docker version 17.9. If a later version is available select the later version. The Privileged option gets selected automatically for Docker runtime as it is required to build a Docker image. For Build specification select Use the buildspec.yml in the source code root directory as shown in Figure 7. The Buildspec name is buildspec.yml by default.

Figure 7. Configuring Environment: How to build

For Certificate select Do not install any certificate. A certificate is not required by CodePipeline, but for a more secure CodeBuild a self-signed certificate could be installed from S3. Next, configure the output artifacts in the Artifacts. Output artifacts are required by CodePipeline Select Type as Amazon S3. Specify a S3 bucket folder name to use. Set the Path as “/”, which creates the folder at the bucket root. Select Namespace type as Node. Select a Bucket name as the Bucket configured earlier as shown in Figure 8.  Select Cache as No cache

Figure 8. Configuring S3 Bucket for Artifacts

In Service role select the option Create a service role in your account if the CodeBuild project is being created for the first time. If the CodeBuild project was created before, select Choose an existing service role from your account. Select the option Allow AWS CodeBuild to modify this service role so it can be used with this build project as shown in Figure 9. For VPC select No VPC. Click on Continue.

Figure 9. Configuring Service Role and VPC settings

In Review, review the Source and Build environment. Scroll down and click on Create to create the CodeBuild project. A CodeBuild project gets created and listed in Build projects as shown in Figure 10.

Figure 10. CodeBuild Project

The service role created by default by CodeBuild does not include some of the required permissions. The service role needs to be modified by adding an inline policy that adds permissions s3:GetObject and s3:PutObject. The inline policy to add is listed:

{
   "Version":"2012-10-17",
   "Statement":[
     {
       "Effect":"Allow",
       "Action":["s3:PutObject","s3:GetObject"],
       "Resource":"arn:aws:s3:::*"
     }
   ]
}

Testing the CodeBuild Project

Test the CodeBuild project before creating and configuring the CodeBuild in a CodePipeline so that if any errors exist they may be fixed. Click on Start build to start the build as shown in Figure 11.

Figure 11. Start build

The Start new build wizard gets started. Click on Start build. The CodeBuild project gets started and the code starts to get built.  When the CodeBuild project gets completed, the Phase details indicate the same as shown in Figure 12.

Figure 12. Phase details and Build logs indicate that the CodeBuild has completed

The Docker image dvohra/node-server generated and updated by the CodeBuild project to Docker Hub is shown in Figure 13.

Figure 13. Docker Image generated and uploaded by CodeBuild  on Docker Hub

Creating a CodePipeline

Having created projects for each of the CodePipeline stages; GitHub code repository for Source, CodeBuild for Build and ECS Fargate service for Staging, next we shall create a CodePipeline. Open CodePipeline Console with URL and click on Get started as shown in Figure 14.

Figure 14. CodePipeline>Get started

The Create pipeline wizard gets started as shown in Figure 15. First, specify a Pipeline name (node-server-fargate) and click on Next step.

Figure 15. Specifying Pipeline Name

Next, configure the Source location. For  Source provider select GitHub as shown in Figure 16.

Figure 16. Selecting Source provider

Next, connect to the GitHub with Connect to GitHub as shown in Figure 17.

Figure 17. Connect to GitHub

Select the GitHub Repository as shown in Figure 18.

Figure 18. Selecting GitHub Repo

Select the repo Branch as shown in Figure 19. Click on Next step.

Figure 19. Source location>Next step

Next, configure the Build. Select Build provider as AWS CodeBuild as shown in Figure 20.

Figure 20. Selecting Build provider as AWS CodeBuild

The subsequent section displayed is based on the Build provider selected. For AWS CodeBuild, a section to provide details about the CodeBuild gets displayed. For Configure your project select  Select an existing project as shown in Figure 21. Select Project name as the CodeBuild project node-server created earlier.

Figure 21. Selecting CodeBuild Project

Click on Next step. Next, configure the Deploy stage of the CodePipeline as shown in Figure 22. Select Deployment provider as Amazon ECS.

Figure 22. Selecting Deployment provider as Amazon ECS

The subsequent section displayed is based on the Deployment provider selected as indicated by the Amazon ECS section in Figure 23.  In the Amazon ECS section select Cluster name as the cluster in which the ECS service to be deployed to is created;  select default.

Figure 23. Selecting ECS Cluster

Select the Service name as the ECS service to deploy to, which is node-server-service, as shown in Figure 24.

Figure 24. Selecting ECS Service

Specify the Image filename as imagedefinitions.json as shown in Figure 25. If omitted, the default Image filename is imagedefinitions.json and the file should be available in the Source code GitHub repo. Click on Next step.

Figure 25. Specifying Image filename

Next, select the Service Role name as shown in Figure 26. A new service role is created in IAM automatically the first time a CodePipeline is created. Subsequently, the Service role gets listed to be selected.

Figure 26. Selecting Service Role

Click on Next step. Review the CodePipeline and click on Create pipeline as shown in Figure 27.

Figure 27. Create pipeline

A CodePipeline gets created as shown in Figure 28. After getting created the CodePipeline starts to run automatically as shown by In Progress status for Source stage.

Figure 28. CodePipeline Source stage in Progress

When the Source stage completes, its status becomes Succeeded as shown in Figure 29. And the Build stage starts to run as indicated by the In Progress status.

Figure 29. Source stage completed and Build stage In Progress

The Build stage also gets completed as indicated by the Succeeded status in Figure 30. The Staging stage starts to run.

Figure 30. Build stage succeeded and Staging stage In Progress

Modifying the Input/Output Artifacts

We had already configured all the CodePiepline stages; why do we need to modify the settings for Input/Output Artifacts? Because for the example ECS application deployed by CodePipeline, the default Input/Output artifacts settings are not as required to run the CodePipeline. By default, the input artifacts to the Staging stage are the output artifacts from the Build stage. For a CodeBuild that only builds a Docker image and uploads the Docker image to Docker Hub or Amazon ECR, no output artifacts are generated by the CodeBuild stage. The input artifacts to the Staging stage need to be set to the output artifacts from the Source stage. Without modifying the settings for the Input/Output artifacts, the Staging stage would fail. The CodePipeline that was started automatically after being created fails. To modify the Input/Output artifacts click on Edit as shown in Figure 31.

Figure 31. CodePipeline>Edit

After modifying the Input/Output artifacts, click on Save pipeline changes as shown in Figure 32.

Figure 32. Save pipeline changes

Running the CodePipeline

To run the CodePipeline after modification, click on Release change as shown in Figure 33.

Figure 33. Release change

In the Release change confirmation dialog, click on Release. The modifications made to the CodePipeline get applied and the CodePipeline starts to run from the start as indicated by the In Progress status for the Source stage in Figure 34. The status for the Build and Staging stages are from the previous run and does not indicate the current status of the stages.

Figure 34. Source stage In Progress

 All stages of the CodePipeline get completed with status Succeeded as shown in Figure 35.

Figure 35. All stages of CodePipeline completed successfully

The previous service task gets stopped and a task based on the revised task definition starts to run as indicated by the RUNNING task status in Figure 36.

Figure 36. New task running

To invoke the new task, find its Public IP as before from the Elastic Network interface for the task. Open URL <Public IP>:8080 in a web browser to invoke the task. The Node server application response is shown in Figure 37.

Figure 37. Node Server application response

Modifying Source Code to Re-Run CodePipeline

What is the advantage of running a CodePipeline, if the ECS service response is the same as when invoking the service directly without a CodePipeline? Typically the source code for an ECS deployed in production would need to be updated periodically, which implies that the Docker image would need to be rebuilt. The Docker image deployed in an ECS service task would also need to be updated. The Docker image needs to be updated without any discontinuation of the ECS based service. With a CodePipeline, a new run of the CodePipeline is started automatically every time source code modifications are made. Without any user intervention, the ECS deployment gets updated when source code changes are made.

To demonstrate, make a slight modification to source code in the GitHub repo, such as make the Hello message in server.js different. Click on Commit changes in the repo. The CodePipeline starts to re-run automatically as shown by the Source stage having completed successfully and the Build stage in progress in Figure 38.

Figure 38. CodePipeline restarted automatically

After the Build stage completes successfully, the Staging starts to run as indicated by the status messages in Figure 39.

Figure 39. Build stage completed and Staging stage started

The Staging also completes successfully. The new task gets started.   Using the Public IP of the new task, open the URL <Public IP>:8080 in a web browser. The new task gets invoked and the Node server response gets displayed as shown in Figure 40. The Node server response is the modified response from server.js. 

Figure 40. Modified Node Server Response from new task

Summary

In this article, we discussed deploying a Docker container application to ECS Fargate.  We have demonstrated updating source code for a Docker image and deploying the new Docker image to a running container service without stopping the container service, all using an AWS CodePipeline.

About the Author

Deepak Vohra is a Sun Certified Java Programmer and Sun Certified Web Component Developer. Vohra has published Java and Java EE related technical articles in  WebLogic Developer's Journal, XML Journal, ONJava, java.net, IBM developerWorks, Java Developer’s Journal, Oracle Magazine, and devx. Vohra has published five books on Docker and is a Docker Mentor.

Rate this Article

Adoption
Style

BT