Agile Testing Challenge

Posted by Mark Levison on May 12, 2010

Sections

Process & Practices

Topics

Collaboration ,

Agile ,

Delivering Quality ,

Agile Techniques

Tags

Testing

Gerald Ford International Airport has a parking lot fee calculator and Matt Heusser noticed that it was buggy. So he issued a challenge to testers around the world: Find the bugs in ParcCalc. Using twitter and his blog a number of people were recruited to test and understand the application. Before reading the rest of the story it's worth taking a few minutes to see how many bugs you find yourself.

Selena Delesie, an Agile Testing Coach, started by trying the boundaries in the date and time fields.

After playing with it for a couple of minutes, I was able to obtain a $6.75 million charge for 1 day of short term parking. …

Some more attempts saw my cost estimate creeping upwards to $2,500,000,000,000,041.00, by simply pushing a limit in the Leave Time field, as seen below.  Notice that I was given a parking duration of an exponential day?  Stranger yet.

KC, a software tester working in the Philippines, noticed that if you try calculating with the default values a page with a PHP error message was displayed before the error was replaced. Other participants in the exercise discovered that this error appeared every time the page was loaded and the page just consumed and ignored the exception.

Various people started playing to see what the highest parking fee they could obtain, the comments following Matt’s original challenge tell an interesting story of collaboration and working together. Along with finding higher and higher values,  new comments built on the previous ones bringing new insights into the underlying problem.

Alain Bohon found some interesting cases that the parking calculator  thought were valid:

• I’ll arrive at 6:am and leaving at 3:am on the same day and I won’t have to pay anything!
• I changed my mind and decided that I want to arrive on 4/4/10 at 27:00 pm and I’ll leave on 4/13/10 at -6:00 am, wow!
• Actually can I arrive at 12.5555:00 and leave at 13.00004:00… I guess I can.

Peter Haworth-Langford noticed that the original exception was generated by a method named mktime(), and by knowing the parameters that it takes he started creating new tests to see if he could find additional bugs.

Dr. Meeta Prakash observed, that the application accepted any input allowing the many out of bounds entries to be accepted.

Matt discovered that while the 7th day is supposed to be free in the economy lot, the way it was coded made it cheaper than free: Six days, 3 hours is Sixty dollars, while Seven days is Fifty-Four dollars! A savings of Six dollars for staying parked an extra day.

Several people discovered that the date picker sometimes disappeared, Matt notes:

The code for the date-picker causes it to pop-up, and do nothing if it is already “up.” Ahh, but there’s a user-interaction problem. If I click the pop-up (to make it appear), then click back on the webform, the date-picker goes under browser. Next time I click for the pop-up to appear, the business logic will say “hey, the pop-up is already up” and do nothing – yet the user won’t see it. This persists, even after refreshing or resubmitting the page.

Several people reported this, mostly noticing the symptom, but with reproduction strategies or models of behavior that were imperfect. … If there had been a price for best bug report, I bet the reproduction strategies would have been better.

Finally Selena observed a number of things along the way:

• No limit on amount of data that could be entered in Date and Time fields.
• No restriction on amount of time that Short-Term Parking applied to.
• In some situations, when Leave Date increased a large amount beyond Start Date, Cost decreased (while all other variables remained the same).
• When entering large numbers (around 10 digits & higher), data in Time fields had a larger impact on Cost than did data in date fields.
• Can enter negative numbers into Date and Time fields, and Cost is calculated.
• Can enter letters into Date and Time fields, and Cost is calculated.
• Can enter symbols into Date and Time fields, and Cost is calculated.
• I suspect you could enter php script into fields and do something interesting (possible security issue?), but my php skills are non-existent these days.
• Found a sort of boundary in each data entry points in Date & Time fields.  For each point X in (X:X, X/X/X) an additional digit added resulted in calculator producing an error, and then an additional digit beyond that resulted in a lower Cost than two attempts ago.
• Can get free parking over course of a day, provided Start Date = Leave Date, and Start Time > Leave Time where:  Start <=11:59, Leave >= 0:59 (both am, or both pm).  This reports negative days parked.

My maximum Cost? $5,014,609,699,998,504.00 (for 2.08942070833E+14 Days, 14 Hours, 47.4666666667 Minutes)

The takeaway:

• Working collaboratively generated more insights than anyone person on their own.
• It's good exercise for Testers to practice sharpening their skills on something that isn’t their day to day focus.
• Developers could read about the problems encountered and consider if similar issues might be found in their own code.

1. Back to top

   Alternative Hosting for ParkCalc

   by Mark Levison

   I was reminded that Adam Goucher provided alternative hosting for ParkCalc that means you don't have to play on server controlled by Homeland Security.
   
   Cheers
   Mark Levison
   Agile Pain Relief Consulting

   Reply

• Older >