BT

Perceptual Testing for Safer Continuous Deployment

by Grischa Ekart on Jun 20, 2013 |

 

Brett Slatkin from Google presented a method for visual regression testing at Velocity 2013 in Santa Clara. Perceptual diff compares screenshots of a release's webpages with its previous versions and detects changes at pixel level.

According to Brett, perceptual diffs fill an important gap in automated testing. Continuous Deployment relies on a multitude of automated tests to increase confidence in the quality of code and deployments. In addition, manual testing is necessary to prevent unforeseen errors. These tests are costly, repetitive and at times ineffective. A lively example was a visual artifact (a pink pony put on a page for testing purposes) that found its way into production, undetected by all QA checks.

Perception diff uses a headless browser (PhantomJS) for webpage rendering and screenshot generation. These images are compared with visual diff. The result is a highlighted depiction of the differing areas, easily recognizable to the human eye.

The method has been presented at last year's Velocity Ignite sessions and has been since used for deployments at Google Consumer Surveys to increase confidence in Continuous Deployment. Brett reported that following difficult to find regressions have been caught: layout errors, number formatting and wrong sorting order. He admitted that false positive results, like time stamps, have to be masked out. Perceptual diffs work well for relatively static pages. Dynamic pages need to be preloaded with identical data sets.

An open source version called Depicted (dpxtd) can be found on github. It wraps the perceptual diff into a workflow that can be integrated into the deployment process.

As an aside, Brett reported positive effects of increased confidence in the deployment process. Shorter deployment cycles lead to increased motivation. Higher confidence in successful deployment empowers juniors to start deploying and therefore shorter ramp-up phases.

 

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT