Continuous Integration with MSBuild and Jenkins – Part 2
This is the second part of a two-part article on Continuous Integration. If you haven't already, please check out Part 1 of Continuous Integration with MSBuild and Jenkins first.
Jenkins was created by Kohsuke Kawaguchi in 2004 and was originally known as Hudson. Sun, Kawaguchi's employer at the time, was providing support for Hudson by 2008. After Oracle's purchase of Sun in 2009, disagreements arose between Oracle and Hudson's developer community. In 2011, due to Oracle's claims to the Hudson trademark, the Hudson developer community renamed the project Jenkins, and continued their work on the project without Oracle's interference.
At first glance, Jenkins seems like a Continuous Integration tool for Java-based projects. There are jobs designed for Maven projects, there are default plugins intended for Java-based project, not to mention that plugins are written in Java.
However, Jenkins is a remarkably flexible tool to build practically any kind of project by combining any source-code management system with any build system. We will take advantage of this flexibility and use Jenkins to pull our changes from a Mercurial repository to build our project using MSBuild. To begin, we'll need to download and install Jenkins and install the Mercurial and MSBuild plugins to get Mercurial and MSBuild support within Jenkins.
Setting Up Jenkins
Grab the installer for Jenkins at the official website. The Windows installer is extremely simple and will install Jenkins as a Windows service. By default, you'll access Jenkins at http://localhost:8080 , so make sure nothing else is running at that port when you first start Jenkins.
Once you startup Jenkins, you'll want to install the necessary plugins. Click the "Manage Jenkins" link and then the "Manage Plugins" link. Click the "Available" tab to view the plugins available to install. You'll need a working internet connection to view the contents of this page. Use the filter to find the Mercurial and MSBuild plugins and install them by clicking their checkboxes. Confirm the plugins were successfully installed by navigating to the "Installed" tab. You may see a message telling you Jenkins needs to restart to complete the plugin installation. Allow Jenkins to restart and navigate to the installed plugins list to confirm that the Mercurial and MSBuild plugins were successfully installed.
We'll need to do some further configuration for Mercurial so Jenkins will know where to find your Mercurial installation. Head back to the "Manage Jenkins" page and click the "Configure System" link. Look for the "Mercurial" section. If you don't find the Mercurial section then the Mercurial plugin was not properly installed. Click the "Add Mercurial" button. Within this section, you'll need to define the name, installation directory, and executable of this Mercurial installation. Give the installation whatever name is convenient for you. As for the Installation directory, give it the directory where the Mercurial executable (hg.exe) is located. Finally, for the executable, type in "INSTALLATION/hg.exe". The word installation will be substituted for the installation directory you've provided above.
Creating a Jenkins Job
Next navigate back to the Dashboard by clicking the "Back to Dashboard" link and click the "New Job" link. Here you'll be presented with a list of job types, select the "Build a free-style software project," give your job the name "HelloCI-RunUnitTests," and click OK.
You'll now be at the job configuration page. There are quite a few options available here and they generally have helpful descriptions available when you click the small help icons to their right. We'll be configuring two settings for now; we'll let Jenkins know where to locate our source code repository and how to build our project using MSBuild.
Find the "Source Code Management" section and select Mercurial. Type in the name of Mercurial installation you defined in the "Configure System" page in the "Mercurial Version" textbox. Then type in the URL of your Mercurial repository in the "Repository URL" textbox. The reposotory URL can be a file system path as well. Finally, pass in the branch you'd like tracked in the "Branch" textbox.
Next, scroll down to the "Build" section and click the "Add build step" button. You should be presented with a list of build step types including "Build a Visual Studio project or solution using MSBuild". If you don't see this option, the MSBuild plugin wasn't properly installed.
After clicking the link, type in the name of the build script in the "MSBuild Build File" textbox (HelloCI.msbuild). We want Jenkins to execute our "RunUnitTests" Target so if you haven't specified that "RunUnitTests" Target in the DefaultTargets attribute, you can pass in the /target switch by typing it in "Command Line Arguments" textbox. Just type in "/t:RunUnitTests" and that should do it (/t is short for /target).
Save your build job by clicking the "Save" or "Apply" buttons at the botton. This build job should pull and compile our project and run the unit test suite whenever it's triggered. Let's trigger a build manually to confirm everything is set up correctly. Head back to the dashboard and you should see your build job in the center of the screen. Click the name of the build job to be taken to the job's page. On the list of links to the left, click the "Build Now" link to trigger a build. Jenkins will then schedule a build job and will begin to execute the job. Underneath the list of links should be "Build History" list which will show a summary of all builds for this job. You should see your first build job begin to execute and a progress bar begin to fill. Also, a small circular icon representing the status of the build should be visible. If it's blinking, then the build is in progress. When the icon stops blinking, it will be red or blue. If it's red, the build failed while blueindicates a successful build.
If your Jenkins job was able to access your Mercurial repository, find your HelloCI.msbuild build script in your repository and execute the "RunUnitTests" Target successfully to completion; the icon should turn blue. You have successfully completed your first Jenkins build. If the build failed, click the timestamp of the build in the "Build History" list and click the "Console Output" link to view every command Jenkins executed and the resultant output. Use the console output to figure out what went wrong.
After your build succeeds, the next step is to get Jenkins to automatically start a build whenever you push new code into your repository. There are a few ways to accomplish this. You could have Jenkins periodically start a build. This is wasteful if there was no new code pushed in the specified time period. Another option would be to have Jenkins poll the Mercurial repository occasionally to check if there was any new code pushed in. The downside to this approach is when you push new code in you'll need to wait until the poll period elapses to have the new build started. You could have Jenkins poll every minute so the wait wouldn't be very long, but there is a more elegant solution. Add a post-commit hook to the Mercurial repository so whenever the repository receives new code, it informs Jenkins of the fact and Jenkins in turn starts a build immediately.
This approach requires you to add the hook within your repository settings file located within the .hg folder (if it doesn't exist, just add a text file named .hgrc). Within it, you'll need to get Mercurial to visit the following URL http://localhost:8080/job/JOBNAME/build?delay=0sec (where JOBNAME is replaced with the name of your build job) so Jenkins is notified to start a new build. Unfortunately, on Windows there is no command like Linux's wget, which can make HTTP requests. I've personally used a Ruby one-liner to accomplish this goal. You can also use a Powershell script to create an instance of System.Net.WebClient and call the DownloadString method on the above mentioned URL. I'll leave the implementation of this approach as an exercise to the reader.
If you'd rather not use this approach, you can always use the polling technique. To do this, return to the build configuration page. Under the "Build Triggers" section, select "Poll SCM" and type in how often you'd like to poll the repository in the "Schedule" textbox. You'll need to type in the schedule using cron-style syntax. To poll the repository every minute, type in "1 * * * *". Use the help icon to the right of the "Poll SCM" text to guide you in setting up your schedule.
You can have Jenkins activate jobs based on successful completion of other builds. You can create build pipelines, where a successful completion of one job can trigger other job. Jobs that activate other jobs in the pipeline are upstream to the jobs they activate and jobs that are activated by other jobs are downstream to the jobs that activate them.
Build pipelines allow for all sorts of scenarios; having more time-consuming tests executed when the unit test suite passes, running some static code analysis tools, or deploying the build to staging or production environments. To demonstrate this capability, we'll use Jenkins to automatically start up our application on a web server whenever it successfully builds.
To accomplish this goal, we need to do only three things; trigger this job whenever our HelloCI-RunUnitTests job successfully builds, copy the build artifacts from the HelloCI-RunUnitTests job and finally start the web server. Before you begin, you need to install the Copy Artifacts plugin. Head back to the Manage Plugins page and follow the same routine as the earlier plugins to do so. Don't forget to restart Jenkins when prompted.
Before creating a new job, we need to tell the HelloCI-RunUnitTests job to save certain build artifacts our new job will need. Go back to the job configuration page of the HelloCI-RunUnitTests job and add an "Archive the artifacts" post-build action under the "Post-build Action" section (below the "Build" section). You'll find a textbox labeled "Files to archive". You want to archive everything within the BuildArtifacts folder your build script created as well as the packages folder so you have the access to the NuGet packages you may need to run your application. To specify that you want a particular folder and its contents, you'll need to type in the folder name followed by a forward-slash and two asterisks. Separate different files and folders you may need by commas. In our case, to get the BuildArtifacts and packages folders, type in "BuildArtifacts/**,packages/**".
Next, create a new Jenkins job and name it "HelloCI-StartWebServer". In the job configuration page, select "Build after other projects are built" in the "Build Triggers" section and specify the earlier job as the project to trigger this job.
You now need to add a build step so the first thing this job will do is copy the save artifacts from the earlier job. Do this by using the "Copy artifacts from another project" build step (which should be available after installing the Copy Artifacts plugin). All you need to specify is the earlier job as the project whose artifacts will be copied.
Finally, add a build step to start the web server. To do this, add an "Execute Windows batch command" build step. We'll use Visual Studio's web server, Cassini, to run our application. Cassini can be found at "C:\Program Files (x86)\Common Files\microsoft shared\DevServer\10.0\WebDev.WebServer40.EXE". Running Cassini is pretty simple, just pass it the path where the web application is rooted using the path parameter and an optional port number if you don't want to, or can't, use the default port 80.
The "Execute Windows batch command" build step requires a command that will execute at same directory the BuildArtifacts and packages directories were copied to. Add the following command into the textbox labeled "Command":
"C:\Program Files (x86)\Common Files\microsoft shared\DevServer\10.0\WebDev.WebServer40. EXE" /port:9876 /path BuildArtifacts/_PublishedWebsites/HelloCI.Web/.
The above command will start Cassini, point it at the application that has been copied from the earlier project, and run the server on port 9876.
All the configuration should now be ready. Try making a change to your application and committing your changes to the repository the HelloCI-RunUnitTests job is tracking. Jenkins should notice the new code, run the HelloCI-RunUnitTests job. If the code compiles and the unit test suite completes successfully, Jenkins should then run the HelloCI-StartWebServer job that copies the artifacts from the HelloCI-RunUnitTests job and start Cassini on port 9876. If it all worked out, you've successfully setup your automated Continuous Integration system.
There's so much more to cover about MSBuild and Jenkins. You can set them up to upgrade from Continuous Integration to Continuous Deployment. All you need is a build job capable of deploying your build artifacts to your deployment target. For ASP.NET applications, this could be as simple as a Jenkins job copying the build artifacts to a certain directory. You can then have that Jenkins job activated whenever a build passes all the tests suites so new features are automatically released to production by Jenkins.
If you're not ready to take it that far, you go forward by using the Promoted Build plugin to promote a build for release. Promoting a build simply means marking a build as promoted when it passes a certain criteria. The criteria for promotion could be when a build is manually approved (when a button is clicked) by one or more individuals. With the Promoted Build plugin, changes could be automatically released onto the development server so all new features are centrally located. A team lead could then promote a particular development build to a staging server. The QA team or other project stakeholders can then sign off (by promoting the staging server build) and the staging server build can be automatically deployed to production by Jenkins.
John Ferguson Smart has authored an excellent book on Jenkins entitled "Jenkins: The Definitive Guide". Besides a thorough briefing on Jenkins' features, the book also gives a great tutorial on how to setup Jenkins to create the build pipeline described above. The book also explains how to secure your Jenkins installation, monitor its resource usage (e.g. disk space in use), distribute your builds to slave machines, and even setup a notification system so you'll be informed whenever a build starts, succeeds, or fails. The book, published by O'Reilly, is available for free, and is a must-have if you're just getting started with Jenkins.
Having an automated build and deployment system is just as important as a strong testing culture and the use of version control in professional software shops of today. I hope that you've gained enough familiarity on how to set up a Continuous Integration system that you can integrate in your own systems.
You may worry about the time to set everything up; don’t be. Set everything up piece-by-piece as you find the time. Eventually, you'll have it all set up and then you can begin to enjoy the fruit of your labor. You will notice bugs earlier as Jenkins displays a broken build, you'll have a systematic deployment process and won't have to remember all the steps it takes to successfully deploy your system. Most importantly, you'll have more time to do useful things like build value-adding features to your application, begin working on new projects, or perhaps being able to go home without having to worry about an upcoming release.
About the Author
Is "Continuous Deployment" as simple as stupid?
I can hardly believe that InfoQ could publish an article like this.
And I can also tell that the Author's knowledge about continuous deployment is of a meagre and unsatisfactory kind, cause anyone has solid experience about deployment won't say "All you need is a build job capable of deploying your build artifacts to your deployment target".
Mustafa Saeed Haji Ali
After having read the first part of the article introducing MSBuild and this follow up with Jenkins, I would hope the path to Continuous Deployment would seem more doable to those who haven't done it before. Sorry if it wasn't made clear that this was an introductory article for those who haven't used MSBuild and a CI tool before.
Mustafa Saeed Haji Ali
However, I do believe its a good start for those trying out Continuous Deployment. Start out by automating one part of the deployment procedure like copying artifacts and slowly work your way towards a fully automated solution that can handle more complicated scenarios.
The reason I decided to publish this introductory series was to show our readers who don’t already use CI as part of their Application Lifecycle that this doesn’t have to be a daunting process. We often talk about ‘why’ continuous integration is important, but we rarely talk about ‘how’ to make it happen.
If you or someone else is interested in writing an article on advanced continuous integration techniques by all means feel free to contact me at email@example.com.
I see your point and I also believe that it's a good start for the readers who'd like to try Continuous Deployment, but my point is, we cannot tell them CD is simple as stupid just because they don't have enough knowledge to understand it. It's lying.
Post-commit hook to Team Foundation Server
Is it possible?
P.S. Great tutorial!
I am configuring Mercurial plug-in in Jenkins. My problem is I cannot get hold of the hg.exe file. Where is it located exactly in the pc?
Also when setting the directory I have something like "INSTALLATION/bin/hg.exe", which is not exactly as you specified. Should I change it in what you suggested?