Git, Gerrit Review and Jenkins or Hudson CI Servers
This article explains how to set up Git, Gerrit and Jenkins/Hudson for team-based code review systems, as espoused in my Gerrit for iOS developers and Gerrit for Java developers presentations (and my someday... discussion, where I first argued the case for using this). The examples used assume you're using OS X or Linux, but you can run on Windows as well if you want to.
Setting up Git
Git is available on most packaging systems already, but there are installers available from the Git homepage. For Windows, the best bet is MsysGit. Note that if you've got the Apple developer tools installed (for Xcode 4) then this comes with Git binaries already. If you have problems, there are a number of very good guides at help.github.com.
Since all Git commits have an author and email address, you need to set up your name as follows, if you haven't done so already:
$ git config --global user.name "Alex Blewitt" $ git config --global user.email "Alex.Blewitt@example.com"
You should be good to go with a Git repository. Gerrit will automatically scan Git repositories at initialisation, which is slightly easier than setting them up afterwards, so putting the Git repositories in before initialising Gerrit makes sense.
If you haven't got an existing Git repository, you can create one suitable for use:
git init --bare /path/to/gits/example.git
Gerrit is available from the downloads at http://code.google.com/p/gerrit/, and exists as a WAR file. There's good documentation available (currently, 2.2.1 is the latest, but this works with 2.1.7 as well) along with the installation guide.
Gerrit needs a database (to store the review information) as part of its runtime. Currently supported databases include H2, PostgreSQL and MySQL. By default, it will use H2 which needs no additional setup.
Note that Gerrit 2.2.x is moving the project configuration, rights and other metadata into Git storage, so that they are accessible and versionable through Git. This transition will continue for other types of metadata, including review notes, in the 2.2.x streams. Please see the release notes for more information.
To initialise Gerrit, run
java -jar gerrit.war init -d /path/to/location to install the runtime in the given path.
If run from an interactive terminal, you are asked various questions, such as:
- Location of Git repositories [git]
- Import existing repositories [Y/n]
- Database server type [H2/?]
- Authentication method [OPENID/?]
- SMTP server hostname [localhost]
- SMTP server port [(default)]
- SMTP encryption [NONE/?]
- SMTP username
- Run as [you]
- Java runtime [/path/to/jvm]
- Copy gerrit.war to /path/to/location/bin/gerrit.war [Y/n]
- Listen on address [*]
- Listen on port 
- Download and install Bouncy Castle [Y/n]
- Behind http reverse proxy [y/N]
- Use SSL [y/N]
- Listen on address [*]
- Listen on port 
Most of these can be left as their default values. However, a few are worth noting the behaviour of.
- Location of Git repositories is where the git repositories live. This defaults to the 'git' directory underneath the installation location, but can be in a different location (e.g.
/var/gitsor similar). It's worth populating this directory with your example first, because when Gerrit starts, it will scan this directory for new projects to add.
- Listen on address is useful for hosts with multiple addresses (e.g. IPv4 and IPv6) as it allows you to constrain which address it uses. The * means any address on the local host.
- Listen on port is the port number. 29418 is the default Gerrit SSH daemon, and 8080 is the default Gerrit web daemon. However, if you have applications using port 8080 already, you might want to change the second one.
- Authentication method is how you log into Gerrit. OpenID works if you want to hook into an existing authentication provider (e.g. Google Accounts) but for testing purposes – and the ones used in the demos above – you can use
development_become_any_account. Typing a ? will show a list of the available methods.
When Gerrit finishes running, you should be opened into a browser that shows you the main page. The first user to log in becomes the administrator automatically; all subsequent users that log in are non-privileged users. If you chose the
development_become_any_account there's a
Become link on the top of the page, which will take you to the registration/sign-in page.
Registering a user
In order to do anything with Gerrit, you need to have a registered account and an SSH keypair generated. Running
ssh-keygen -t rsa -b 2048 from a command line allows you to generate a keypair, which gets put in your
.ssh directory. There's more information at the GitHub Help page, although if you want more information you can see this blog post I wrote six years ago on SSH keys.
The default will be called
id_rsa (which is the private key) and
id_rsa.pub (which is the public key). Only ever give out your public key, never your private key.
With the key in hand, you can register a new account in Gerrit. Click on the “become” link on the top right, followed by the “New account” button, and put in your name and e-mail as they're known by Git (the one we configured above with
git config). These have to match exactly (including case). You can save changes, and then pick a unique user name (click 'select username' once you've filled it with a name e.g.
Email headaches Gerrit will try and send you an e-mail to verify your mail address, even in
development_become_any_accountmode. Without it, it doesn't like your e-mail address, and without that, you can't push code.
We can hack that shortly, so don't worry if it's not letting you register your mail address just yet.
In the 'add SSH public key' text box, add the key exactly as it is given in the .pub file. If you're on OS X, this is as easy as
pbcopy < ~/.ssh/id_rsa.pub. Remember to click on “Add” to save it.
When you click Continue, you should see you logged into Gerrit's main window. So far so good. Now we can test SSH connectivity.
ssh -p 29418 demo@localhost will try and talk to the Gerrit server, which will say one of three things:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
This isn't nearly as bad as it looks. It just means that there's an old key in your
~/.ssh/known_hostsfile. The lazy way to fix the problem is to just delete this file (which is GitHub's recommendation). The better way is to look for the line which tells you which line the error is, and delete just that line:
Offending key in /Users/demo/.ssh/known_hosts:123
So we need to delete line 123 from the
known_hostsfile, which you can do with any text editor you want. Given a standard UNIX setup, you can do it automatically:
sed -i '' '123d' ~/.ssh/known_hosts
The authenticity of host '[localhost]:29418 ([::1]:29418)' can't be established. RSA key fingerprint is e8:e2:fe:19:6f:e2:db:c1:05:b5:bf:a6:ad:4b:04:33. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[localhost]:29418' (RSA) to the list of known hosts. Permission denied (publickey).
If you get this message, it means that Gerrit doesn't recognise any of the keys you have submitted. By default, ssh will send
id_rsa, but you can ensure this is the case with an entry in the
.ssh/configwith a line
IdentityFile ~/.ssh/id_rsaat the top. You can run
ssh -vwhich will expand on what it is sending:
debug1: Next authentication method: publickey debug1: Trying private key: /Users/demo/.ssh/id_dsa debug1: Offering public key: /Users/demo/.ssh/id_rsa
Assuming it's sending the right key, check that the key is associated with the user in Gerrit. This is at the settings - ssh public keys menu option. If it's not present, click on 'Add Key' and paste in the public key as before.
If you get this message, and Gerrit is still complaining that you are unauthenticated, check the user name matches the username specified in the settings page. If the username is something different, try
ssh -p 29418 username@localhostinstead.
Finally, to verify a specific key, run
ssh -i ~/.ssh/id_rsato explicitly choose which key to use, instead of letting it get picked automatically. If this works, but running without the
-iparameter fails, then the issue is in your
~/.ssh/configfile – you need to make sure that the appropriate
IdentityFileis being selected.
**** Welcome to Gerrit Code Review **** Hi demo, you have successfully connected over SSH. Unfortunately, interactive shells are disabled. To clone a hosted Git repository, use: git clone ssh://demo@localhost:29418/REPOSITORY_NAME.git Connection to localhost closed.
If you see this, Gerrit is working as expected.
Fixing the email address
If you couldn't register an e-mail address in Gerrit earlier, you can do so manually. We'll stop Gerrit, then run the GSQL tool to update the columns appropriately.
$ bin/gerrit.sh stop $ java -jar bin/gerrit.war gsql Welcome to Gerrit Code Review 188.8.131.52 (H2 1.2.134 (2010-04-23)) Type '\h' for help. Type '\r' to clear the buffer. gerrit> select * from ACCOUNT_EXTERNAL_IDS; ACCOUNT_ID | EMAIL_ADDRESS | PASSWORD | EXTERNAL_ID -----------+------------------------+----------+------------------------------------------ 1000000 | NULL | NULL | uuid:ac1b8a08-2dd1-4aa1-8449-8b2994dffaed 1000000 | NULL | NULL | username:demo (2 rows; 23 ms) gerrit> update ACCOUNT_EXTERNAL_IDS set EMAIL_ADDRESSfirstname.lastname@example.org' where ACCOUNT_ID=1000000; UPDATE 2; 5 ms gerrit> select * from ACCOUNT_EXTERNAL_IDS; ACCOUNT_ID | EMAIL_ADDRESS | PASSWORD | EXTERNAL_ID -----------+------------------------+----------+------------------------------------------ 1000000 | email@example.com | NULL | uuid:ac1b8a08-2dd1-4aa1-8449-8b2994dffaed 1000000 | firstname.lastname@example.org | NULL | username:demo (2 rows; 23 ms) gerrit> \q Bye $ bin/gerrit.sh start
Creating a project, cloning and pushing
To get started, we need a project in Gerrit. If it didn't detect the project in your example directory, we can create one subsequently if Gerrit is running.
$ ssh -p 29418 demo@localhost gerrit create-project --name example.git
This will create a project called
example, and initialise an empty repository in the gits location specified above. If you have an existing repository, it won't let you create it with the same name – but you rename it to a temporary name prior to creation and then rename back it will still work.
Having made it available in Gerrit, we can create a clone:
$ git clone ssh://demo@localhost:29418/example.git Cloning into example... warning: You appear to have cloned an empty repository.
We can commit and push to our repository, much like any other Git system:
$ cd example $ echo hello > world $ git add world $ git commit -m "The World" [master (root-commit) 06bf85e] The World 1 files changed, 1 insertions(+), 0 deletions(-) create mode 100644 world $ git push No refs in common and none specified; doing nothing. Perhaps you should specify a branch such as 'master'. $ git push origin master Counting objects: 3, done. Writing objects: 100% (3/3), 217 bytes, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://me@localhost:29418/example.git ! [remote rejected] master -> master (prohibited by Gerrit) error: failed to push some refs to 'ssh://demo@localhost:29418/example.git'
What happened here? Well, Gerrit doesn't want us overwriting any branches directly in the Git repository. Instead, we must push to a different refspec, which gives Gerrit the opportunity to put the code through review. The easiest way to do that is to configure the default refspec for pushes:
$ git config remote.origin.push refs/heads/*:refs/for/* $ git push origin Counting objects: 3, done. Writing objects: 100% (3/3), 217 bytes, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://demo@localhost:29418/example.git * [new branch] master -> refs/for/master
We've pushed a branch, referred to here as
refs/for/master, although the actual named branch
refs/changes/01/1/1, which you can see from the change,1 in Gerrit itself. You'll note that the current branch is the same hash as shown here (in my case,
06bf85e). If we were to push again, instead of overwriting the
refs/for/master, we'll trigger the creation of a new branch
refs/changes/02/2/1. Although it's an implementation detail, the first digits are the last two digits of the change number, and second digits are the change number, and the third digits is the patch set number. So patch set 17 of change 123 would refer to an immutable reference
If we wanted to commit amend this change, Gerrit will create a new review reference, which is not so good. You'll lose the review request comments between the two changes if that happens.
To fix that, we'll update the
commit-msg to add a (Gerrit-specific)
Change-Id. This will allow all subsequent commits of the same change to be associated with the same patch set. Gerrit comes with an implementation; we can just copy that out.
$ cd .git/hooks $ scp -P 29418 demo@localhost:hooks/commit-msg . $ cd ../..
Now, when we commit amend, we'll get a
Change-Id field automatically generated. We can use this to generate multiple patches on our change set. We can fix the current commit to use the same change set by doing an amended commit and putting the one shown in the Gerrit change.
$ git commit --amend -m "Hello World > > Change-Id: I06bf85ed12f370212ec22dbd76c115861b653cf2 > " [master 86a7a39] Hello World 1 files changed, 1 insertions(+), 0 deletions(-) $ git push Counting objects: 3, done. Writing objects: 100% (3/3), 260 bytes, done. Total 3 (delta 0), reused 0 (delta 0) remote: (W) 86a7a39: no files changed, message updated To ssh://me@localhost:29418/example.git * [new branch] master -> refs/for/master
If you now look in Gerrit at change 1 you'll see that there is a second patch set associated with the change. The remote branch
refs/changes/01/1/2 now contains the new commit message (although all the files remain the same).
Normally you won't need to add the
Change-Id in place, as the commit message hook will do it for you automatically.
Reviewing and submitting
To substitute a build process, we'll create a
build.sh script which trivially succeeds. Jenkins/Hudson can then check this out in order to run something. Typically this would be a Maven build process or an
make – to avoid a language-specific environment, we're just using a script which succeeds trivially.
$ cat > build.sh #!/bin/sh echo Pretending to build ... echo done ^D $ chmod a+x build.sh $ git add build.sh $ git commit -m "Adding (dummy) build script" $ git push
The problem is, these changes are still in the review queue for Gerrit. We'll need to go there and approve them. If you go into the change, you'll see the files present and a
Review button. You'll note that there's a
Review button on the change, which lets you review the code. There's no submit button though ...
The reason there's no submit button is because a change needs to be reviewed with a
+2 by default before it can be submitted. Since each person can add a
+1, this means it's not possible to submit with a single person.
We can change this by updating the rules in Gerrit to allow an individual to submit +2. We could change this by allowing an individual to vote +2; we can also do the same with updating the rules to permit a +1.
The easiest way to change it is to go into the project admin and into the all projects access tab (which was renamed from --All Projects-- to All-Projects in Gerrit 2.2.1 and 184.108.40.206). Click on the checkbox next to the “Code review” rule, and change the “Permitted Range” to go up to
+2: Looks good to me, approved. Gerrit also needs a +1 verified, which we'll set up for Jenkins/Hudson as well. Add a new rule with the following:
- Group Name
- Non-Interactive Users
- Reference Name
- Permitted Range
- -1: Fails to +1: Verified
Click on “Add Access Right” to set up this permission.
We'll need to create a new user for Jenkins/Hudson and put it in this group, so we'll go to sign-in page and register a new account,
buildbot. We'll need a new ssh key (call it
id_rsa.buildbot) and then paste in the public key as before.
Finally, sign back in as the administrator ID (the one you created earlier) and go to the Admin - Groups tab. In the list of non-interactive users, add both the
buildbot and (temporarily) the
Having set up the verified rule, we should now be able to go back into the change, in order to mark it as verified.
However, we're still missing the
submit button, which is what we need to merge it on to the branch. Submit rights are a separate set of rights to the verification and review rights, so we have to go back in to the all projects access tab and add an access right as before:
- Group Name
- Registered Users
- Reference Name
- Permitted Range
- +1: Submit
Click on "Add Access Right" to set up this permission.
Now, when we go back into the change, we'll see a
Submit button to the review (if it's been reviewed) as well as a
Publish and Submit on the comments/review page. (Note that you may see a server error when doing a
Publish and Submit if the code review hasn't reached the appropriate level for submission.)
Finally, publish and submit all reviews outstanding (to make sure that the build script is in place) and do a
git pull to ensure that your repository is up-to-date. They should show up in the merged tab.
Note about administration: typically, you won't give 'All Projects Access' rights universally, but rather do it on a project-by-project basis. The 'All Projects' approach is being shown here to make it easy to get up and running, but these can be configured on a per-project basis as well.
Setting up Jenkins/Hudson
The final piece of the puzzle is setting up Jenkins (or Hudson, for preference). Given that they run on port
8080 by default, as does Gerrit, you need to get one to run on a different port number. Changing Gerrit involves re-running the setup procedure, but you can change Jenkins/Hudson by command line.
$ #java -jar hudson-2.0.1.war --httpPort=1234 $ java -jar jenkins.war -httpPort=1234 ...
Jenkins/Hudson can check out Git projects directly, or they can check out via the Gerrit code review. However, the check out doesn't always have the ability to customise the SSH identity (other than the default set mentioned in the
.ssh/config file). It can sometimes be easier to host the Git repositories by anonymous Git protocol, which doesn't need authentication. To do this, you can run:
$ git daemon --export-all --base-path=/path/to/gits
We'll also need to install the Git plugin as well as the Git/Gerrit trigger. To do this, open Jenkins/Hudson on http://localhost:1234 and click on the
Manage Jenkins/Hudson link on the top left, and go into the
Manage Plugins link. Switch into the
Available tab, and then install:
- Gerrit Trigger
Install button is right down the bottom; it will restart in a few moments. The Gerrit Trigger will pull in the Git plugin. Note that there is a Gerrit plugin, which is not the one needed. (If you're using Hudson, and the update site doesn't show any content, then go to 'Advanced' and click on the 'Check now' at the bottom.)
Now we're ready to start configuring Jenkins/Hudson to do the work. To start with, we'll set up a CI job that looks out for changes to the (merged) Git repository. Do the following steps:
- Click on the 'New Job' at the top left. It will ask for a name (e.g.
Example) and the type; in this case, the
- Select Git as the SCM type (if it's not shown, it means that you need to install the Git plugin above). The URL will be
- For the build triggers, check the box next to
Poll SCMand put
* * * * *in it. This is a crontab-like format, which in this case equates to checking the repository every minute.
- Scroll down to
add a build stepand select
execute shell. Put in
$WORKSPACE/build.sh(note: if your project name has a space, you may need to surround this with quotes e.g. "$WORKSPACE/build.sh")
- Finally, click on
Saveand the project should be created.
This has created a build which checks out
master, and if it changes, executes
build.sh. Clicking on the
build now link should check out the code, run the script, and report success. (If this fails, check the build log for more information and resolve before going further.)
Integration with Gerrit
We've now got Jenkins/Hudson building
master whenever it changes. However, the final piece of the puzzle is getting it to build whenever a new review is submitted.
Create another job with the following:
- Advanced (below URL of repository) Refspec
- Advanced (above repository browser) Choosing strategy
- Build Triggers
- Gerrit Event
- Gerrit Project
- Path - ** - Path - **
This will allow the project to be triggered by Gerrit whenever a new event occurs. However, we have one final piece of the puzzle to hook up – we have to tell Jenkins/Hudson which Gerrit server to listen to.
Manage Jenkins/Hudson tab, an entry
Gerrit Trigger will have been added. Click on this and add the following information:
- SSH Keyfile password
First, save it by clicking on the “Save” button below. Then, check this works by clicking the
Test Connection button. If this succeeds, click on “Restart” at the bottom of the gerrit trigger to pick up the changes, or just restart Jenkins/Hudson.
If all has gone well, then you should be able to go back to the main page where it has a
Query and Trigger Gerrit Patches link on the left. Click on this, and enter
is:open to see any open changes, and
is:merged to see any merged changes. Once that is done, check the box and hit
Trigger Selected to fire off a build against that specific change.
If you are using Hudson 2.0.0 and Gerrit 2.2.0, you might find that there are
java.lang.reflect.InvocationTargetException errors in the Hudson console. This was a problem with the newly upgraded Gerrit Trigger, but upgrading to Hudson 2.0.1 fixes that issue.
Putting it all together
You should now be able to make a change in a local clone of the repository, push it to Gerrit, and have Jenkins/Hudson build it automatically for you.
To test out a failed case, edit the
build.sh script and put
exit 1 at the end. Commit and push that to Gerrit, and you should see Jenkins/Hudson change the state to failed, because now the build script is returning a non-zero code. Amend commit to return
exit 0, push, and you'll find the build is marked as verified.
Finally, if building iOS applications with
xcodebuild, then piping the build through ocunit2junit.rb script, which converts the SenTest assertions into a form which is understandable by Jenkins/Hudson, such that the failed assertions can be printed on the output build log.
About the Author
Dr Alex Blewitt is founder of Bandlem Limited, and works at an investment bank in London, but still finds the time to catch up with the latest OSGi and Eclipse news. Despite having previously been an editor for EclipseZone and a nominee for Eclipse Ambassador in 2007, his day-to-day role involves neither Eclipse nor Java. In what little time he has left over, he spends with his young family and has been known to take them flying if the weather's nice. You can follow Alex on Twitter at @alblue or his blog at alblue.bandlem.com.
- may move gerrit trigger configuration in front of the setting up build job, since it is the gerrit trigger plugin to trigger the build, then pass the build refspec ($GERRIT_REFSPEC) to git plugin, therefore git plugin notice which branch (refs/changes/xxx) to build.
- It could be possible to click one build and check the parameters for all values passed from gerrit trigger plugin to git.
- testing connection -> save configuration -> restart could be better for configuration on gerrit
- openid is little tricky, better to mention LDAP authentication (which is also common for enterprise) as well.
Glad to see more and more team start to use gerrit and jenkins
Re: Glad to see more and more team start to use gerrit and jenkins
more complicated workflows
Re: more complicated workflows
Re: Great article
It is possible to set up this only using Gerrit Trigger if you want to. As for authentication options; the best place is to direct you to the Gerrit manual which has those steps - I chose the development purely for convenience and in order to allow people to experiment with the setup. In real organisations, it is likely that the setup would involve more detailed configuration which was outside the scope of this article.
Re: Glad to see more and more team start to use gerrit and jenkins
You can also write your own authentication handlers or wrap it in front of an Apache/Tomcat instance that provides the authentication layer if you want to.
For what it's worth, this article setup was tested on both Mac OSX and Windows.
Re: more complicated workflows
Re: more complicated workflows
Randy Shoup Jul 03, 2015