Best.ly offers support for A/B-testing mobile applications without the need for code changes and resubmitting to an app store. Currently, there is only a native library for the iOS platform but libraries for the Android eco-system are already in development and will be available soon.
A/B-testing is often used in marketing and advertising to optimize the click-through-rate of customers. Therefore, for example a web site is published in two versions (A and B), that differ in one certain aspect. Test users are divided into two groups, each using one version of the site. At the end of the test, the conversion rates for both versions of the site are compared to measure which version was more successful. The winning version will then be published for all users.
After signing up, Best.ly lets developers of mobile apps administrate different applications online. For each application, there can be an arbitrary number of experiments and within an experiment an arbitrary number of variations. An experiment represents the component in the application which needs to be tested, a variation is one of the different versions of the component.
Integration of Best.ly with an iOS application is done via a CocoaPod library. The different variations for the experiment are declared as blocks, the Objective-C construct for a lambda or closure. Which block actually gets executed is chosen by the Best.ly library depending on the administered settings for the experiment.
During the test, Best.ly allows to monitor user behavior in real-time on a dashboard. The weights for the variations can be edited during runtime and eventually, the desired variation of the experiment can be committed. After committing a variation, the experiment has finished and all users will then use the chosen variation. Since Best.ly selects the variations to be executed, the application code does not need to be changed for adjusting variation weights or committing a variation and there is no need for resubmitting the app to the app store.
Pricing depends on the number of monthly users of the tested application. Up to 200 monthly users are free, 2.000 are $19, 20.000 are $79 and 200.000 users are $399 per month. All plans include multi-variant testing and an unlimited number of experiments.