BT

Swift’s Benchmarking Suite is Now Open Source

by Sergio De Simone on Feb 12, 2016 |

Apple has open sourced Swift’s benchmarking suite, a key piece in tracking Swift performance and catching performance regressions when adding new features to the language.

Swift’s benchmarking suite is a collection of Swift source files that implement test suites and benchmarking helper functions, plus a number of Python scripts that implement a test harness and facilities for metrics comparison.

Apple suggests running the benchmarking suite before committing any changes that may impact performance, and before submitting a pull request, which is done by passing the --benchmark flag to build-script, Swift’s build automation script is in charge of basic tasks such as building a Swift compatible LLDB, building for a given platform, running tests after build etc. This should help catch potential performance regression. In order to make sense of the benchmark results, it can be useful to first take a baseline benchmark on Swift’s master branch, then re-run the benchmark on the development branch, whose results will be automatically compared to the previous run’s.

By default, only Swift benchmarks for OS X are compiled, but additional platforms can be specified on the command line like this:

$ swift/utils/build-script --ios --watchos --tvos

When using the benchmark driver, developers can control the number of loop iterations in each test sample and the number of samples to take for each test, e.g.:

$ ./Benchmark_O --num-iters=1 --num-samples=1

As Apple engineer Luke Larson wrote, at the moment, Swift’s benchmarking suite includes 75 benchmarks, but Apple is encouraging contributions for new benchmarks, to helper libraries, and for general improvements. An important requirement for contributions, Larson says, is that they be covered under the same license as Swift. So ports of benchmarks covered under other licenses cannot be accepted.

Swift’s benchmarking suite is not currently integrated with the recently announced Swift continuous integration system, which builds and runs tests for OS X, iOS, and x86_64 Ubuntu Linux. Based on Jenkins, Swift CI system aims to make it feasible to support more platforms and architectures coming from the Swift development community. Swift CI system enables the execution of tests within a pull request, i.e., before committing to master, and notifies failures to the responsible developer. According to Larson, though, Swift’s benchmarking suite will be integrated within Swift CI system in the future.

Rate this Article

Relevance
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss
General Feedback
Bugs
Advertising
Editorial
Marketing
InfoQ.com and all content copyright © 2006-2016 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT