BT

Thoughts on Unit Testing vs Ad-hoc Testing with Asynchronous Code

| by Jonathan Allen Follow 631 Followers on Aug 18, 2014. Estimated reading time: 1 minute |

One of the interesting features of the Task + async/await pattern is that you can easily decorate the results of any operation. Lucian Wischik of Microsoft shows how to take advantage of this in order to make your end-to-end testing more robust.

Using Mr. Flakey is as simple as adding “.Flakey()” to the end of each await statement in any Windows 8 or Windows Phone 8 application. Dim r = Await http.GetStringAsync(uri).Flakey()

Once enabled, each async call will cause a dialog to appear. You can either press “ok”, which returns the result as normal, or “fail” to simulate a network or server issue.

Why do this kind of ad-hoc testing instead of just writing a comprehensive set of unit tests? Lucian explains,

Unit tests. In my experience unit tests aren't very good for distributed algorithms. They take a fair bit of effort to set up (e.g. to mock the network), and they're doomed to not be comprehensive enough, and in my experience they just aren't very good at finding distributed bugs. I suspect it's because coders' brains think mostly along the "success" path of their algorithm, and aren't good at envisaging all the failure paths.

(Think: when you look at concurrent code, are you the person who spots race conditions just by looking at it? Can anyone on your team do this? Have you ever written a unit test that uncovered a race condition? I haven't.)

Ad-hoc testing. But what's great at finding bugs is interactive ad-hoc testing. If the UI to simulate failure is easy enough, you can almost set your mind on auto-pilot and randomly click here and there, and you quickly find places where the app doesn't behave right.

I was struck by Leslie Lamport's work on the "Temporal Logic Checker" in the 1990s -- a way to automatically explore that huge exponential space of possibilities (similar to what we face in distributed algorithms). What he found was that "automated random walks" through the space of possibilities ended up in practice finding just about all the bugs, and was vastly more efficient than an exhaustive test of every single possibility.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss
BT