Parse has announced Push Experiments, a feature that allows to conduct A/B tests for campaigns based on the use of push notifications. Push Experiments will allow to evaluate different messaging strategy and choose the one with the highest open rate, say Parse.
Push notifications are a mechanism often used to improve user engagement in an app, although with varying results. A push notification will show up as a pop-up message to its receivers, conveying a fixed message. Critical factors in determining a push notification effectiveness to induce the user to launch the app are the message content, as well as the time when the message is received.
Push Experiments' goal is to allow performing several push messaging experiments while holding all external factors constant. This means allowing only the message that's being tested to change. External factors that could affect how a given message turns out in terms of open rate is, e.g., getting featured in the media between one experiment and another, say Parse, and this would certainly affect the use rate of the app.
The way Push Experiments allow to execute A/B testing is the following:
- You allocate a subset of your users to two test groups and sends a different message to each group.
- In Parse push console, you can see in real time which version resulted in more push opens, along with other metrics such as statistical confidence interval.
- Later, you can send the message that performed better to the rest of the user base.
Push Experiments can also be used to evaluate at which time of the day sending a message is more effective.
You can also check InfoQ "A/B testing" section for more great content about online field experiments.