Heads up: Our Ideas Factory has been refreshed, levelled up, and grown-up into Alphero Intelligence. Some of our old posts are pretty cool tho'. Check this one out.
- A/B testing tools for Native have been available in the market for a while, and our team wanted to get a better idea of what they offer.
- We have a wide range of capability to test different layouts and functionalities and learn what works best for users.
- It’s easier to implement when considered upfront, and we wish more clients would prioritise this to enhance the performance of their apps.
The Ideas Factory is a place for us to test and evaluate tools internally - without clients having to pay for these investigations.
As many of our clients reach maturity and stability with their app, we start looking into ways of making things better and building on the experience. Sometimes, small changes to a screen can make things clearer and drive more engagement, and it’s great when we can pair up little changes to analytics to measure how changes impact users. This practice can complement user interviews and surveys, quantifying discoveries from UX research and answering some questions much more objectively and effectively.
A/B testing has been common practice in the web space for years, but until recently it has been somewhat of a fringe practice with native apps. Early last year we set ourselves on an #ideasfest (ideation session) to investigate different A/B testing frameworks for native apps, seeking to understand:
- What sorts of variants are easy or difficult to test?
- How easy/difficult is the technical implementation and what are the potential gotchas?
- What reporting/analytics capabilities are there for product owners (both quality of underlying algorithms and any UI)?
- What are the ongoing operational considerations?
With that in mind, we picked up five popular A/B testing frameworks and went to work on a few of our clients’ apps!
Our team discovered that each framework has different approaches, and the level on which we can test simple or complex variants can be quite different. Most frameworks include UI editors where you can make minor changes without the need for redeploying the app, but these changes are quite limited and any tests that include more complex UI changes will require a redeployment.
As far as reporting goes, all frameworks do a good job in their own ways, and we can work with product owners to compare the offerings of each service and find the framework that’s right for them.
During our investigations it became clear that retrofitting A/B testing into an existing application can vary depending on the application and what you are testing. But once a direction is settled and the code is cleaned up there aren’t any ongoing maintenance impacts.
We do believe that the others still have merit, and this investigation provided us with a lot of insight to be able to choose the best option depending on the app size, design features and data appetite.
Connor Moody, a Mobile Developer at Alphero give us his take:
"Out of the five frameworks we tested, our team reckon Firebase and Leanplum were top of the list, providing the best offering around analytics and options for testing."