BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles Managing Build Jobs for Continuous Delivery

Managing Build Jobs for Continuous Delivery

Lire ce contenu en français

Continuous Delivery (CD) facilitates the path of an evolving product from development through to production. Continuous Integration (CI) plays an important part of this process and defines the software development process, residing at the beginning of the CD process.

There is a wealth of information in books and on the Internet describing CI processes, however with a large amount different CI tools available there is little information on the build jobs which compile and test the code that are central to the CI process.

A typical CI process is as follows: The developer manually builds and tests source code on their own machine. They will then commit their modifications to a source control management system. A build tool will subsequently run jobs which compile and test the code. The built artifact is then uploaded to a central repository ready for further deployment and testing.

Hence, it is the role of jobs in the CI tool to manage the continuous modifications of the source code, run tests and manage the logistics of artifact transportation through the deployment pipeline.

The number of jobs in a build tool can range from just a few to perhaps several thousand, all performing various functions. Fortunately, there is a way to manage all of these jobs in a more efficient manner.

Automatic Creation of Build Jobs

So why should we set up a facility to automate the creation of other jobs?

Build tool user guides describe how to create build jobs but they do not often explain how to manage them in any great detail, despite the functionality being available in the tool to do so.

Normally the developer will know how their application is built and so take sole responsibility for the creation and configuration of the build job. However, this process has some disadvantages:

  1. Software architectural constraints: Applications are likely to be built differently from one another, depending upon architecture. The developer may interpret how an application is built differently from another developer and so the build configuration will slightly differ from one job to another. Hence, a large number of jobs become unmaintainable if they are all configured differently.
  2. Human factor: Manually creating a job introduces the  risk of making mistakes especially if a new job is created by copying an existing job and then modifying that job.
  3. Job timeline control: There is generally no record kept of job configuration for every build, i.e. If a job configuration is changed it could break previous builds.

So considering the points above, if developers are 'left to their own devices' to create build jobs without a consistent method for creating jobs, then we are going end up with a poorly maintained CI build system and a headache to manage. This defeats the purpose of how a CI system should be set up; lean and maintainable.

Most build tools in fact have the facility to create, maintain and backup build jobs automatically usually via API scripts. However, these features, despite mentioned in the build tool user's manual, are often overlooked and not implemented.

Advantages for automating the creation of build jobs

There are several advantages of using a master build job to automatically create multiple jobs in order to facilitate the CI build process.

  1. All build job configurations are created from a set of job templates or scripts. These can be placed under configuration control. Creation of a new job is then a simple matter of using the template or running the scripts. This can cut job creation times significantly (many minutes to a few seconds). Hence, Job configuration changes are made easier; configuration changes to the build job template ensures that all subsequent new jobs created will inherit the new changes.
  2. Consistency with all existing build job configurations implies that globally updating all configurations via tools or scripts is more straightforward than with a cluttered inconsistent system.
  3. The developer does not necessary need detailed knowledge of the build tool to create a build job.
  4. The ability to automatically create and teardown build jobs as part of an Agile CI build lifecycle.

Let's discuss the points listed:

Point 1: Configuration Control of Job Configuration

It is good practice that every component of a CI system should be placed under configuration control. That includes the underlying build systems and the build jobs, not just the source code.

Most build tools store build job configuration as a file on the master system. These files are set to a common XML format and are normally accessible via the front end interface directly through a REST API of some kind. Some build tools have a built-in feature that can save job configuration as a file to any location.

Since the job configuration can be saved as a file it can be stored in a configuration management (CM) system. Any job configuration change can be recorded through either modifying the file directly, then upload the file to the build tool or modifying the job configuration from the build tool, save the file and upload it manually to the CM system. The method carried out depends upon how easy it is to access the job configuration file directly from the build tool.

Another important reason for storing job configuration in a CM system is that in the event of a catastrophic failure and all job configurations are lost, the build system can be recovered relatively quickly and all jobs, build logs and job history restored back to their last known good state.

Point 2: Job Maintenance

Of course it goes without saying that maintaining a thousand jobs of different configurations would be a major headache, which is why it is so important to standardise job configurations. If a change is required that impacts on a large number of similar jobs, then writing scripts or creating special tasks to modify these jobs would be straightforward.

However, having a thousand jobs in a build system is not the best strategy for CI.  We will discuss this further under Point 4.

Point 3: Developer Control

Although developers are encouraged to work autonomously, building their applications using a centralised build system, they need to work within guidelines outlined by the build process.

A developer will expect fast feedback of their code changes and so not want to waste a lot of time messing around with build tools and job configuration. Providing developers with a one button 'self service' style solution to set up build jobs automatically, will help them achieve their development task objectives more quickly.

Point 4: A Lean CI System

Although software developments teams adopt Agile methods in developing their products, the build tools they work with are not necessarily used in the same way, i.e. a properly configured build tool should not contain thousands of long lived build jobs. Ideally, jobs should be created on demand as part of a build CI lifecycle. As long there is the means to recreate any build job from any historical job configuration and that build relics are retained, i.e. logs, artifacts, test reports etc., then only a handful of jobs should exist in the build tool.

Of course an on-demand job creation-teardown solution may not be an achievable goal, but the point is that the number of jobs in a build tool should be kept to a manageable level.

Day-to-day Development

Build management governs the tools and processes necessary to assist developers in day-to-day activities with regards to build and continuous integration. Utilities can be set up which empowers development teams to create build jobs and provide flexibility in job configuration but still be governed by best software development practices.

The CI Job Suite

Consider typical tasks that developers carry out during the course of day to day development:

  • Build a baseline
  • Release the baseline
  • Build a Release branch (often combined with releasing the baseline)
  • Create a development branch, build the branch
  • Run Integration Tests

Best development practices aside, i.e. develop on baseline as much as possible, hence few or no development branches, software development teams generally follow the tasks outlined above by running a suite of build jobs in the build tool.

Now this is where automatic creation of jobs adds huge value to the software development process; if a developer wanted to created four to five jobs to cover all of the development tasks to manage their day to day operations, it would take them a significant amount of time and effort to create the jobs manually. This is in comparison with automatically creating jobs in a fraction of the time at a push of a button.

When to implement a self-service job creation solution

Implementing a self-service solution for creating build jobs is applicable for the following cases:

  1. Initialisation of new 'green-field' projects (those that have never been built before).
  2. Projects that already exist but never had any build jobs set up for them. i.e. building manually from a developer's own machine.

There are also cases where implementing a self-service job creation solution may not applicable:

  1. If a product only has a few jobs, i.e. large monolithic applications, then it may be considered not of value to the business to spend a considerable amount of time and effort designing, testing and rolling out an automated create job process for just those few projects. However, as the business expands, the product architecture will become more complex as more development streams are set up to manage the extra workload, hence the number of jobs in the build tool will inevitably increase.
  2. Projects with existing jobs: Two scenarios can be considered when dealing with existing jobs in the build tool.
    • Delete the existing jobs and recreate them with the job suite creation utility, thereby losing all historical build information and logs.
    • Keep the existing jobs and attempt to modify their configuration to match the job configuration created by the job suite creation utility.

In any case, there is always going to be some overhead with maintaining legacy build jobs.

Setting Up a Self-Service Job Suite Creation Facility in a Build Tool

We have talked about the advantages of automatically creating build jobs, so now let's discuss how we can implement such a feature in a build tool. The principle should be the same for building applications for any language I.e. Java, .NET etc...

Introducing the Job Creation Build Form

The build form is a means to pass information from the build tool directly to another feature or backend scripts that perform the job creation tasks.

Ideally, as much project information should be provided upfront. This will reduce the amount of any additional effort required later to add extra configuration to the newly created build jobs. Information such as project title, location of source code and specific build switches are normally required.  Hence, at a push of the 'Build' button the jobs are created just a few moments later.

Build Tool Implementation:

We shall now discuss how such a job creation facility can be implemented in a build tool. Two tools for which I have had personal experience of setting up job suite creation facilities are Hudson and Anthill Pro 3.

Hudson / Jenkins

For clarity we will just talk about Hudson, but the information is equally relevant for Jenkins.

The Hudson CI tool has become very popular for Build Managers looking to minimise their development resource budget and do not want to be tied down with expensive licences. Hudson defines its build job configuration in XML format. A typical job config XML file is accessible from the top level page of the job url, i.e. http://hostname/job-name/config.xml

Most of the sections in the config file are self-explanatory. Sections are added by plugins in the config file only when they are used in the Hudson GUI.

These config files can be 'templated' by replacing specific project information in them with tokens. Several templates can be used to carry out different tasks, i.e. Checkout of source code from different SCMs e.g, Subversion or GIT.

So now we need to link the create job build form to the scripts. The diagram below shows the workflow:

Hudson Job Creation Workflow

The build form passes the information to the scripts which substitutes the tokens in the templates with the relevant information. The config files are then uploaded to the Hudson tool via an API. More details about the API can be found in Hudson by adding the string ‘api’ to the url in the browser, i.e. http://hostname/job-name/api.

An example of a Hudson API create job upload command is shown below:

curl –H Content-Type:application/xml –s –data @config.xml ${HUDSONURL}/createItem?name=hudsonjobname

Hudson Create Job Form

The jobs can be manually added to a new view.

Hudson Job View

A few points to note:

  1. If the Hudson security section is configured, ensure that the 'anonymous' user has 'create job' privileges otherwise the upload will fail authentication.
  2. If Hudson tabs are used to view specific groups of jobs, it will not be possible to automatically place any of the newly created jobs under those tabs, except for the generic 'all' tab (or any tab that uses a regex expression).  The Hudson master configuration file will require updating as well as a manual restart of the Hudson interface. Once the jobs are created they can then be manually added to the relevant tab if so desired.

Scripts can be written in any language, such as ANT, Perl, Groovy etc., so it is a relatively straightforward set of tasks to create scripts in order to carry out the steps outlined above.

A typical ANT script is shown below:

<project name="createjob" default="createHudsonjobsConfigs">

<!-- Ant script to update hudson job config.xml templates to create new jobs in hudson -->

<!-- Get external properties from Hudson build form -->
  <property name="hudsonjobname" value="${HUDSON.JOB.NAME}" />
  <property name="scmpath" value="${SCM.PATH}" />
  <property name="mvngoals" value="${MVN.GOALS}" />

<!-- ...do same for rest of properties from the hudson form -->
...
...
  <property name="hudson.createItemUrl" 
value="http://hudson.server.location/createItem?name=" />

<!-- Include ant task extensions-->
  <taskdef resource="net/sf/antcontrib/antlib.xml"/>

<target name = "createHudsonjobsConfigs" description="creates new config.xml file from input parameters">

<mkdir dir="${hudsonjobname}"/>

<!-- loop through each job template file replacing tokens in the job-templates with properties from Hudson -->
  <for list="CI-build-trunk,RC-build-branch" param="jobName">
      <sequential>
      <delete file="${hudsonjobname}/${configFile}" failonerror="false"></delete>
      <copy file="../job-templates/@{jobName}-config.xml" tofile="${hudsonjobname}/${configFile}"/>
      <replace file="${hudsonjobname}/${configFile}" token="$HUDSON.JOB.NAME" value="${hudsonjobname}"/>
      <replace file="${hudsonjobname}/${configFile}" token="$SCM.PATH" value="${scmpath}"/>
      <!-- ...do same for rest of tokens in the job template -->
      ...
      ...
      <antcall target="configXMLUpload">
        <param name="job" value="@{jobName}"></param>
      </antcall>
      </sequential>
  </for>
</target>

<!-- contruct the job config upload command -->
<target name="configXMLUpload">
  <echo>curl -H Content-Type:application/xml -s --data @config.xml ${hudson.createItemUrl}${hudsonjobname}-${job}</echo>
  <exec executable="curl" dir="${hudsonjobname}">
    <arg value="-H" />
    <arg value="Content-Type:application/xml"/>
    <arg value="-s" />
    <arg value="--data" />
    <arg value="@config.xml" />
    <arg value="${hudson.createItemUrl}${hudsonjobname}-${job}"/>
  </exec>
</target>
</project>

Below is a typical tokenised Hudson job template:

     <?xml version='1.0' encoding='UTF-8'?>
     <project>
       <actions/>
       <description>Builds $POM.ARTIFACTID</description>
     ...
     ...
       <scm class="hudson.scm.SubversionSCM">
         <locations>
           <hudson.scm.SubversionSCM_-ModuleLocation>
           <remote>$SCM.PATH/trunk</remote>
           <local>.</local>
           <depthOption>infinity</depthOption>
           <ignoreExternalsOption>false</ignoreExternalsOption>
         </hudson.scm.SubversionSCM_-ModuleLocation>
       </locations>
     </scm>
     <assignedNode>$BUILD.FARM</assignedNode>
   ...
   ...
     <builders>
       <hudson.tasks.Maven>
         <targets>$MVN.GOALS</targets>
         <mavenName>$MVN.VERSION</mavenName>
         <usePrivateRepository>false</usePrivateRepository>
       </hudson.tasks.Maven>
     </builders>
   ...
   ...
   </project>

CI-build-trunk-config.xml Hudson Job Template

Anthill Pro

Anthill Pro is a licensed build and continuous integration tool from Urban Code Inc. which provides the user almost complete custom control of build jobs via a GUI and an extensive API library.

The latest incarnation from UrbanCode is uBuild, which is essentially a cut down version of Anthill and is geared primarily towards the continuous integration building of products rather than its parent product which was originally designed to manage to entire continuous delivery pipeline. uBuild does not have the API capability that Anthill has but it is possible to create a plugin that can do a similar thing.

Build jobs in Anthill are called 'Workflows' which essentially are a pipeline of individual job tasks executed in a specific order (serial or parallel).

Anthill's API scripting language is Beanshell which is based on standard Java methods.

Beanshell scripts are stored within the Anthill graphic user interface and called from within a job task. The latest version of Anthill allows custom-built plugins which can be developed with any programming language.

An Anthill job task is created using API calls which essentially 'constructs' all the necessary functions of that workflow. The script can be particularly long as every aspect of the workflow configuration has to be populated with data. Just like Hudson, build job information is collected via a build form which executes a master workflow, calling on the Beanshell scripts to create the relevant workflow.

Anthill Job View

Anthill Create Workflow Form

A typical Java Beanshell script is below.

private static Project createProject(User user) throws Exception {
         // get the values from the buildlife properties
        String groupId = getAnthillProperty("GROUPID");
        String artifactId = getAnthillProperty("ARTIFACTID");
        String build_farm_env_name = "linux-build-farm";

        String branchName = "branchName";

        Workflow[] workflows = new Workflow[3];
        // Set up Project
        Project project = new Project(artifactId + "_" + branchName);

        // determine whether the project already exists and is active
        boolean isProjectActive;
        try {
            Project projectTest = 
getProjectFactoryInstance(artifactId + "_" + branchName);
        isProjectActive = projectTest.isActive();
      } catch (Exception err) {
          isProjectActive = false;
      }

      if (!isProjectActive) {
          project.setFolder(getFolder(groupId, artifactId));
          setLifeCycleModel(project);
          setEnvironmentGroup(project);
          setQuietConfiguration(project);

          String applications = getAnthillProperty("APPS");
          // create project properties
          createProjectProperty(project, "artifactId", artifactId, false, false);
          // set the server group for the workflow and add environment properties
          addPropertyToServerGroup(project, build_farm_env_name, applications);

          project.store();

...
...

    // Create the CI Build workflow
            String workflowName = "CI-build";
            workflows[0] = createWorkflow(
                    project,
                    workflowName,
                    perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId,
                    perforceTemplateFor(branchName, groupId, artifactId, workflowName),
                    "${stampContext:maven_version}",
                    build_farm_env_name,
                    "CI-build Workflow Definition"
                    );

            // add trigger url to commit-build branch
            addRepositoryTrigger(workflows[0]);

    // Create the Branch workflow
            workflowName = "Branch";
            workflows[1] = createWorkflow(
                    project,
                    workflowName,
                    perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId,
                    perforceTemplateFor(branchName, groupId, artifactId, workflowName),
                    "From ${stampContext:maven_version}",
                    build_farm_env_name,
                    "Branch Workflow Definition"
                    );

   // Create the Release workflow
           workflowName = "Release";
           workflows[2] = createWorkflow(
                   project,
                   workflowName,
                   perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId,
                   perforceTemplateFor(branchName, groupId, artifactId, workflowName),
                   "${stampContext:maven_release_version}",
                   build_farm_env_name,
                   "Release Workflow Definition"
                   );

...
...

        } else {
            // project already exixts
            String msg = "Project " + artifactId + "_" + branchName + " already exists - check project name and pom";
            throw new RuntimeException (msg);
        }
        return project;
    }

Example CreateProject Java Beanshell script

Note: It is also possible to create a self contained package of Java Beanshell scripts such as a jar file which is placed in a folder on the Anthill Pro application server. Direct calls to classes within the jar file can then be made from a shell task within the application. This works well especially because the scripts can be unit tested before being applied in a live development environment.

Other Tools

So, as long there exists the ability to create jobs via an API or a template feature, the implementation described here can be similarly applied to other build tools such as Thoughtworks Go, Jetbrains Teamcity and Atlassian Bamboo to name but a few.

Ultimate Build Tool Agility: Implementing the Lean Build Tool philosophy

An advantage mentioned earlier in this article discussed the concept of automatically creating on –demand CI projects and then tearing them down after completion. This would significantly reduce the amount of build jobs in the build tool.

I have yet to implement an agile lean method for automatically creating and tearing down jobs on a build tool, so the technical challenges to implement such a task have yet to be realised. However, everything that has been discussed here can lead to implementing such a solution. In fact the Atlassian build tool Bamboo has the feature to detect branch creation from a main baseline and then automatically create an appropriate job to build it.

Summary

We have discussed how to set up facilities to create other jobs that form part of the continuous delivery strategy for a business.

  • An automated approach to creating jobs in a build tool should be adopted as part of everyday software development operations.
  • Automatic creation of jobs saves time and allows the developer to get on with more important tasks.
  • Consistent build job configuration leads to easier job maintenance and drives consistent development practices across a company.
  • Automatic job creation can ultimately lead to a true Agile adoption of software development right down to the build tool level.

About the Author

Martin Peston is a Build and Continuous Integration Manager for Gamesys, the UK’s leading online and mobile gaming company owners of the popular bingo brand Jackpotjoy and social gaming success Friendzys on Facebook.

Martin has 15 years of cross-sector IT experience, including Space, Defence, Health and Gambling. He has dedicated around half his career to build management and is passionate about bringing Continuous Delivery best practices to software development companies.

In his spare time, Martin is a keen amateur astronomer and is author of "A User's Guide to the Meade LXD55 and LXD75 Telescopes” published by Springer in 2007.

Rate this Article

Adoption
Style

BT