Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Thoughts on Unit Testing vs Ad-hoc Testing with Asynchronous Code

Thoughts on Unit Testing vs Ad-hoc Testing with Asynchronous Code

One of the interesting features of the Task + async/await pattern is that you can easily decorate the results of any operation. Lucian Wischik of Microsoft shows how to take advantage of this in order to make your end-to-end testing more robust.

Using Mr. Flakey is as simple as adding “.Flakey()” to the end of each await statement in any Windows 8 or Windows Phone 8 application. Dim r = Await http.GetStringAsync(uri).Flakey()

Once enabled, each async call will cause a dialog to appear. You can either press “ok”, which returns the result as normal, or “fail” to simulate a network or server issue.

Why do this kind of ad-hoc testing instead of just writing a comprehensive set of unit tests? Lucian explains,

Unit tests. In my experience unit tests aren't very good for distributed algorithms. They take a fair bit of effort to set up (e.g. to mock the network), and they're doomed to not be comprehensive enough, and in my experience they just aren't very good at finding distributed bugs. I suspect it's because coders' brains think mostly along the "success" path of their algorithm, and aren't good at envisaging all the failure paths.

(Think: when you look at concurrent code, are you the person who spots race conditions just by looking at it? Can anyone on your team do this? Have you ever written a unit test that uncovered a race condition? I haven't.)

Ad-hoc testing. But what's great at finding bugs is interactive ad-hoc testing. If the UI to simulate failure is easy enough, you can almost set your mind on auto-pilot and randomly click here and there, and you quickly find places where the app doesn't behave right.

I was struck by Leslie Lamport's work on the "Temporal Logic Checker" in the 1990s -- a way to automatically explore that huge exponential space of possibilities (similar to what we face in distributed algorithms). What he found was that "automated random walks" through the space of possibilities ended up in practice finding just about all the bugs, and was vastly more efficient than an exhaustive test of every single possibility.

Rate this Article