Continuous Integration with MSBuild and Jenkins – Part 1
You or one of your colleagues just finished writing some code and pushed the changes to the team's repository. But wait; what if the new code broke the build? What if it causes a compilation error? Or broke some tests in the test suite? Or what if the code didn't achieve the minimum value expected for some code quality metric?
One unreliable way to avoid these possible issues is to hope all members of the team are diligent enough to make sure these kinds of mistakes don't happen. But if they do happen you want to learn about them as soon as possible. The best way to do this is to have the build verified whenever any new code is pushed into the repository, which is what Continuous Integration can do for you.
There are a variety of tools available for Continuous Integration. One of the most popular is a Java-based tool called Jenkins. Jenkins allows you to create build jobs via a web interface which consists of a series of build steps. You could configure Jenkins to look for all the things mentioned in the scenario above. You could even take it further and use Jenkins to deploy your application automatically or with a single click.
Developed by an ex-Sun employee, Jenkins clearly shows its Java-based roots. However, it isn't just useful for Java-based projects. Jenkins can handle your Continuous Integration needs for your PHP, Ruby on Rails, or .NET projects. For .NET based projects, you're going to need to be familiar with one more tool to make use of Jenkins; MSBuild.
Visual Studio uses MSBuild to build your .NET projects. All MSBuild requires is a build script and a target within the build script to execute. Your *.csproj and *.vbproj files are MSBuild scripts.
In this article, we'll be writing our very own MSBuild script from scratch so with a single command we'll be able to delete the previous build artifacts, build our .NET application, and run our unit tests. Then we're going to setup a Jenkins build job to pull any new code changes from our repository and run our MSBuild script. We'll also setup a second Jenkins build job to monitor the first job. Whenever our first job completes successfully, it will copy the relevant build artifacts and launch a web server with our new code.
We'll be using an ASP.NET MVC 3 application as the sample application. The application is the default application you get when you create a new ASP.NET MVC 3 application and select the "Internet application" template. We'll also be using a unit test project so we have some tests to work with. You can find the source code for the application here.
MSBuild is a build system for Visual Studio introduced in .NET 2.0. MSBuild runs build scripts to accomplish a variety of tasks, most notably compiling your .NET projects into executables or DLLs. Technically the compiler (csc, vbc, etc.) does the heavy lifting of producing EXEs and DLLs. MSBuild calls the compiler internally and does the rest of the steps required to produce the necessary output files (copying references marked CopyLocal, running the pre- and post-build events, etc.)
MSBuild does all this by executing a series of tasks specified in an MSBuild script. An MSBuild script is an XML file with a root Project element that has the MSBuild xml namespace specified. All MSBuild files must also contain one Target. A MSBuild Target is a collection of Tasks that MSBuild undertakes to accomplish a certain goal. A Target may contain 0 Tasks but all Targets are required to have a name.
Let's create the “Hello World” of MSBuild scripts to make sure we've got everything setup as expected. I'd suggest using Visual Studio to get some IntelliSense support, but you could use a text editor instead as the IntelliSense isn't all that useful and we are just writing XML here. Create a new XML file and name it "basics.msbuild". The msbuild extension is a convention we'll use for identifying MSBuild scripts and is not necessary. Add a Project element as the root element of your XML file and give it the XML namespace http://schemas.microsoft.com/developer/msbuild/2003 by setting the Project element's xmlns attribute like so:
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> </Project>
Next, add a Target element within your Project element and name it "EchoGreeting" by setting the Target element's Name attribute.
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <Target Name="EchoGreeting" /> </Project>
That's it. The above is a valid MSBuild script that should run. It won't do anything but we'll use it to verify we can run MSBuild scripts. To run the above build script, we'll use the MSBuild executable located within the .NET framework installation folder. Open up the command prompt and verify you have the .NET framework installation folder in your path by executing "MSBuild /nologo /version". You should see the version number of your MSBuild printed on the screen. If not, add the .NET framework installation folder to your path or simply use the Visual Studio Command Prompt which already has everything set up.
Navigate to the directory you've saved the above build script in and execute the script by calling the MSBuild executable and passing in the file name of the build script as an argument. Here's what executing the above script looks like on my local machine:
C:\>msbuild basics.msbuild Microsoft (R) Build Engine Version 4.0.30319.1 [Microsoft .NET Framework, Version 4.0.30319.269] Copyright (C) Microsoft Corporation 2007. All rights reserved. Build started 8/2/2012 5:59:45 AM. Build succeeded. 0 Warning(s) 0 Error(s) Time Elapsed 00:00:00.03
After executing the build script, MSBuild first displays a startup banner and copyright message (which can be suppressed with the /nologo switch). Next the build’s startup time is printed and the build is run at this point. Since our build script doesn't do anything, nothing else is shown, the build is considered a success, and the time elapsed is printed. Now let's add a Task to our EchoGreeting Target so our build actually does something.
<Target Name="EchoGreeting"> <Exec Command="echo Hello from MSBuild" /> </Target>
The above modified EchoGreeting Target now contains an Exec Task. An Exec Task runs the command specified in the Command attribute. Try running the build script again. You ought to see more text appear on the screen. For the most part, MSBuild is being verbose with its output. You could edit it down by setting the /verbosity switch to minimal. In any case, you can see that MSBuild has successfully echoed our text. Before moving on, let's add another Target to the Project element.
<Target Name="EchoDate"> <Exec Command="echo %25date%25" /> </Target>
The above target echoes the date onto the screen. The command to do so is actually "echo %date%" but the "%" character has special meaning in MSBuild so has to be escaped. Escaping characters requires substituting them for the ASCII code for the character in hexadecimal with a preceding "%" character. MSBuild will only run the first Target specified in the Project element. To run a different Target instead, you'll have to pass in the Target name via the "/target" switch (or "/t" for short). You can also instruct MSBuild to execute multiple Targets by separating the Target names with semicolons.
C:\>msbuild basics.msbuild /nologo /verbosity:minimal /t:EchoGreeting;EchoDate Hello from MSBuild Thu 08/02/2012
A More Useful Build Script
That's enough of the test build script. Let's use MSBuild to build an actual project. Grab a copy of the sample application source code or create your own ASP.NET application to work with. Add an MSBuild script and name it exactly the same name of your solution or project but with the ".msbuild" extension, then add the Project element with the MSBuild namespace on the Project element as before.
Before we start with this build script, let's list the things we want this script to do:
1. Create a BuildArtifacts directory
2. Build the solution and place the build artifacts (DLLs, EXEs, content, etc.) inside the BuildArtifacts directory
3. Run the unit tests suite
Since the sample application is called HelloCI, we're going to call our build script HelloCI.msbuild. Add the Project element with the MSBuild xmlnamespace as before. Now let's add our first Target: Init.
<Target Name="Init"> <MakeDir Directories="BuildArtifacts" /> </Target>
The above Target makes use of the MakeDir Task to create a new directory called BuildArtifacts at the same location as the build script. Try running the build script. You should find the directory was successfully created in the same directory as the build script. If you run the build script again, MSBuild will simply skip the MakeDir Task since the expected directory already exists.
Next let's add a Clean Target. The Clean Target will remove the BuildArtifacts directory and any files within it.
<Target Name="Clean"> <RemoveDir Directories="BuildArtifacts" /> </Target>
The above should be pretty self-explanatory if you've understood the Init Target. Try running the Clean Target. The BuildArtifacts directory should be removed. Next, let's get rid of a bit of duplication. In both the Init and Clean Targets we've hardcoded the name of the BuildArtifacts directory. If we wanted to rename the directory in the future, we'd need to do so in two places. We can make use of MSBuild Items or Properties to avoid such issues.
Items and Properties in MSBuild are very similar but have one subtle difference. Properties are simply key-value pairs and can be set when running the build script by using the /property switch. Items are more powerful than Properties since they allow complex values to be stored within them. We won't be using any complex values but we'll need to use Items to get some extra metadata from some Items like the full path of a file.
We'll use an Item to store the name of the directory we'll be working with and then modify the Init and Clean Targets to reference the Item.
<ItemGroup> <BuildArtifactsDir Include="BuildArtifacts\" /> </ItemGroup> <Target Name="Init"> <MakeDir Directories="@(BuildArtifactsDir)" /> </Target> <Target Name="Clean"> <RemoveDir Directories="@(BuildArtifactsDir)" /> </Target>
Items are defined within an ItemGroup element. There can be multiple ItemGroup elements within a Project element allowing you to group related items together. This becomes useful when you've got a build script with a lot of Items. Within the ItemGroup element is our BuildArtifactsDir element and we include the BuildArtifacts directory using the Include attribute. Don't forget the trailing slash when specifying the BuildArtifacts directory. Finally, we reference the BuildArtifacts directory in our Targets using the @(ItemName) syntax. Now, if we wanted to change the name of the directory, we would simply change the value of Include attribute in the BuildArtifacts Item.
We should resolve one more issue before moving on. Currently, if the BuildArtifacts directory exists, the Init Target does nothing. This means any existing files are left on the disk when calling the Init Target. It would be better to have the BuildArtifacts and all existing files within it removed and have the directory recreated whenever the Init Target is called. We could manually call the Clean Target before calling the Init Target every time, but a simpler solution is to add a DependsOnTargets attribute to the Init Target telling MSBuild to run the Clean Target every time the Init Target is executed.
<Target Name="Init" DependsOnTargets="Clean"> <MakeDir Directories="@(BuildArtifactsDir)" /> </Target>
Now we don't have to call Clean before calling Init; every time we call Init, MSBuild will first run Clean. As the attribute name implies, we can have multiple Targets run before a Target. When specifying multiple Targets, separate them by semi-colons.
The stage is now set to compile our application and place the compilation output into our BuildArtifacts directory. We'll be creating a Compile Target, which depends on the Init Target. The Compile Target will call another instance of MSBuild to compile the application. We'll also be passing the BuildArtifacts directory as the directory our compilation output should be placed.
<ItemGroup> <BuildArtifactsDir Include="BuildArtifacts\" /> <SolutionFile Include="HelloCI.sln" /> </ItemGroup> <PropertyGroup> <Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration> <BuildPlatform Condition=" '$(BuildPlatform)' == '' ">Any CPU</BuildPlatform> </PropertyGroup> <Target Name="Compile" DependsOnTargets="Init"> <MSBuild Projects="@(SolutionFile)" Targets="Rebuild" Properties="OutDir=%(BuildArtifactsDir.FullPath);Configuration=$(Configuration);Platform=$(BuildPlatform)" /> </Target>
There are quite a few things happening here. First, we've added another Item to the ItemGroup called "SolutionFile" that references our solution file. It's good practice to not hardcode values within the build script; use an Item or Property instead.
Next, we created a PropertyGroup and added two Properties within it; Configuration and BuildPlatform. We've set the values to these properties to "Release" and "Any CPU" respectively. However, Properties can be set when executing an MSBuild script by using the /property switch (or /p for short). We've used the Condition attribute on both Properties to tell MSBuild to only set the Properties as the values we've defined if they have not been set already. What we've done is set a default value for both Properties.
We then defined a new Target named Compile which depends on the Init Target. Our Compile Target has within it an MSBuild Task. By doing so, we've called another instance of MSBuild from our MSBuild script. We've specified the Project we want the inner MSBuild Task to execute. Here we could specify another MSBuild script (which are also known as projects), or we could pass in a .csproj file which is also an MSBuild script. Instead, we've passed in the solution file for our HelloCI application. Solution files are not MSBuild scripts but MSBuild is capable of parsing solution files. We've also specified the Target the inner MSBuild Task to execute to be the "Rebuild" Target, which has been imported into the .csproj files within our solution. Finally, we set three Properties for our inner MSBuild Task:
The output directory the inner MSBuild Task should save the compilation output.
The Configuration to use when building (Debug, Release, etc.)
The Platform to compile for (x86, x64, etc.)
When setting each of the above Properties, we've used the Items and Properties we've defined earlier. For the OutDir Property, we passed in the full path of the BuildArtifacts directory. To do so, we've used the %(Item.MetaData) syntax. The syntax should seem familiar and look like how you would access a property on an object in C#. MSBuild gives you access to certain MetaData for any Item you create like the FullPath or ModifiedTime. This MetaData is not always useful since you could have Items which are not files.
As for the Configuration and Platform Properties, we've reference the Configuration and BuildPlatform properties we've defined by using the $(PropertyName) syntax. Here's a list of reserved Property names which are off-limits. Try not use any of them when defining your own Properties.
One more thing to note is that the use of Properties allows us to build using a different Configuration or BuildPlatform Properties without changing our build script; we just pass in different values when running MSBuild by using the /property switch. So running "msbuild HelloCI.msbuild /t:Compile /p:Configuration:Debug" will build the projects in the Debug Configuration and running "msbuild HelloCI.msbuild /t:Compile /p:Configuration:Test;BuildPlatform:x86" will build the projects in the Test and x86 Configuration.
Running the Compile Target now should compile both projects in the solution and place the output in the BuildArtifacts directory. We now have one final Target to add before wrapping up our build script.
<ItemGroup> <BuildArtifacts Include="BuildArtifacts\" /> <SolutionFile Include="HelloCI.sln" /> <NUnitConsole Include="C:\Program Files (x86)\NUnit 2.6\bin\nunit-console.exe" /> <UnitTestsDLL Include="BuildArtifacts\HelloCI.Web.UnitTests.dll" /> <TestResultsPath Include="BuildArtifacts\TestResults.xml" /> </ItemGroup> <Target Name="RunUnitTests" DependsOnTargets="Compile"> <Exec Command='"@(NUnitConsole)" @(UnitTestsDLL) /xml=@(TestResultsPath)' /> </Target>
In the above, we've added 3 new Items to our ItemGroup; NUnitConsole, which points to our NUnit console runner that will run our unit tests project; UnitTestsDLL, which points to the unit tests DLL the unit tests project produces; and TestResultsPath, which we'll pass to NUnit so it will place the test results in our BuildArtifacts directory.
The RunUnitTests Target makes use of the Exec Task which will run the command specified in the Command attribute. If one test fails in our test suite, the NUnit console runner will return a non-zero return value. The non-zero return value will signal to MSBuild that something went wrong and will mark the build as failed.
We now have a robust build script capable of removing old artifacts, compiling our projects, and running our unit tests in one command.
C:\HelloCI\> msbuild HelloCI.msbuild /t:RunUnitTests
We could also set a default Target for our build script so we wouldn't have to specify a Target. Just add a DefaultTargets attribute to the root Project element specifying the RunUnitTests Target as the default Target.
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="RunUnitTests">
You can even add your own custom Tasks to your MSBuild scripts. As an example, check out the AsyncExec project. It allows you to execute commands asynchronously. For instance, if you added a Target that starts a web server, using the Exec command would cause the build to wait unit the web server shuts down before it reported the build complete. Using the AsyncExec command will allow you to run commands without causing the build to wait for them to complete.
You’ll find the completed build script in the sample application repository.
In my next article, I will discuss setting up Jenkins, so instead of having to run 1 command manually to build our projects, we can have Jenkins monitor our source code repository and start a build whenever we push new code to the repository.
About the Author
Mustafa Saeed Haji Ali
I couldn't recommend one over the other as I haven't used Team City enough to comment on it so the only thing I can point out is that Jenkins is free (in both senses of the word) and open-source.
Team city is an alternative to Jenkins - and MSBuild will always be required to describe your build. What exactly is "existing" here?
Alfredo Cavalcanti Segundo
Waiting for the next part. Thanks!
Thanks a lot!
P/S: In the section when you recommended use the commands with params(properties), the syntax used is incorrect, because you should have written using:
For example, use:
"msbuild HelloCI.msbuild /t:Compile /p:Configuration:Test;BuildPlatform:x86"
"msbuild HelloCI.msbuild /t:Compile /p:Configuration=Test;BuildPlatform=x86"
Open source and free only?