BT

Why Gerrit chose Buck

by Alex Blewitt on Oct 31, 2013 |

Today at EclipseCon, Shawn Pearce (maintainer of the Gerrit project) gave a talk on Improve your Java Builds with Buck (slides, additional slides). These gave the reasoning behind why the Gerrit project switched from Maven to Buck, ultimately moving the build to use Buck only from Gerrit 2.8 onwards.

Buck is a build system, loosely modelled on the internal Google build system 'blaze', developed by ex-Google employees who are now at Facebook. Whilst at Facebook, the build system was open-sourced under an Apache licence and is available at GitHub. Buck is based on Python, although it is used primarily for compiling Java in Gerrit.

The Buck language is a DSL which uses Python as its underlying build file. The following defines a Java library called 'printy_lib' which depends on Guava, which is located elsewhere on the file system:

java_library(
  name = 'printy_lib',
  srcs = glob(['src/main/java/**/*.java']),
  deps = [':guava'],
)

prebuilt_jar(
  name = 'guava',
  binary_jar = 'guava.jar',
)

The Gerrit project has added support for resolving JARs from Maven Central, and then materialising them on the local system. The contents of the JARs are identified by a GAV and checksummed with SHA1, which is verified upon download. These extensions aren't available in Buck yet, but they may be contributed in future.

include_defs('//lib/maven.defs')

maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:14.0',
  sha1 = '67b7be4ee7ba48e4828a42d6d5069761186d4a53',
  license = 'Apache2.0',
)

The key advantage of Buck over Maven is that it is fast. This is due to several key advantages; builds are executed in parallel across modules, and the Buck build by default executes 1.25xCPU threads. (Maven and Make also have an ability to run with parallel builds, but still build the modules in parallel). The quoted speed improvements for Gerrit are:

  • Clean build
    mvn package -Dmaven.{javadoc,test}.skip=true  ... 6m50s
    buck build :gerrit                            ... 2m 3s
    buck test --all                               ... 2m 5s
    
  • No-op incremental rebuild
    mvn package -Dmaven.{javadoc,test}.skip=true  ... 4m44s
    buck build :gerrit                            ...    2s
    
  • Buck is FAST; Even faster with buckd running in the background
    ~/buck/bin/buckd
    time buck :gerrit                             ...    0.5s
    

As well as the multi-threaded support by default, Buck speeds up compilation by doing incremental builds and only re-compiling what classes change. Instead of using filesystem timestamps (which may not be reliable on all systems), Buck uses an SHA content match based on the last time the code was compiled to know what the source files have been modified. The hash also includes the hash of the build files, so if the build system is modified then all files are recompiled.

Buck also has detailed knowledge of the Java relationships, so that it knows how to build the relationships between interdependent classes, ensuring that compilation for dependencies is only kicked off if the public API changes, and not if internal methods are changed.

As well as sharing the content for builds on the local system, it's possible to hook up Buck to an Apache Cassandra storage system to permit the libraries to be shared between multiple developers. This acts as a repository mechanism for downloading components which have already been built, or for older versions, and the build system can resolve against these. This permits easy bisecting to find a problem, and instead of having to do a build each time the assets from previous builds can be resolved. As the contents are hashed, the builds are available even if the versions aren't updated or the build hasn't been publicly released.

This increase in speed has enabled more flexible development; given that a build can execute in less than a second (if the Gerrit daemon is running) the development build of Gerrit will detect if it is running in development mode, and kick off a rebuild and reload for every HTTP request. This permits very fast turnaround of patches and development without needing to spend seconds or minutes doing a rebuild and deploy.

Buck is still in development, and perhaps currently lacks some of the community and documentation that Gradle provides. Finally, although Buck is written in Java, it uses Python as its DSL, and the testing is largely done on Mac and Linux machines, which means that Windows support is a second class citizen at the moment.

For more information on Buck, see the FaceBook page describing Buck

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Gerrit vs Maven by Sergey Kabashnyuk

Is there any public source code where I can test the difference?

Frustratig by Happy SpaceInvader

Sound reasons for moving to Buck, I'm sure, and I'm all for progress, but...

... using Buck is a very different experience from Ant, Maven and Gradle, in that it does not have anything in the way of stable releases. Thus, you clone it in Git, build with Ant (yuk) and, according to the Buck installation documentation, soft link the resulting binary straight to /usr/bin.

Then, when you try and build a project, watch out for the .buckversion file, which will then try to upgrade/downgrade Buck to the version it wants. Except that it fails to do this, so you then add a .nobuckcheck file to the project root, understanding that YMMV.

And all for a speed benefit that I would love to see, but I can't because when building gerrit, Buck is failing with all sorts of new errors that I don't recognise.

I'm sure it will be great once I've solved all of these issues I didn't have with Maven, but for now, it gets a slow hand clap.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

2 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT