Product Page Optimization (PPO), or App Store native A/B testing, became publicly available on December 7. And now, with the official release of Google Play’s experiment platform counterpart, both stores now have true A/B testing tools to gather platform-specific learnings and gain a better understanding of users’ preferences.
More tests are yet to come that will explore the history of testing between the two platforms, as well as the extent to which PPO is able to move the needle on App Store conversion rates. For now, let’s recall the initial WWDC21 promise and dive into the test setup in the App Store Connect.
What is Product Page Optimization?
First announced on July 7, 2021 at the Apple Worldwide Developers Conference, Product Page Optimization is the newest native A/B/C/D testing environment in the App Store. Developers were promised that they would be able to test their product page with up to 3 different treatments to identify the most conversion-centric app page creatives.
To put it into numbers:
- 3 product page variations per test
- 3 assets can be changed (icon, screenshots, app preview)
- Up to 90 days to run the test
It was said that all the variations would be reviewed by the Apple team unless they used pre-approved screenshots. Screenshots and app previews could be uploaded and applied independent of the app release, while the app’s icon should be included in the binary of the published app.
The new App Analytics tool was also announced to support A/B testing with impressions, conversion rate, and improvement features included to help compare performance against the original product page throughout the test.
Besides numerous, well-known benefits of A/B testing (e.g., identifying the most engaging features, value propositions, eye-catching designs, etc.), PPO also reveals valuable insights when testing seasonal assets and culturally relevant content for a certain location.
New PPO Details You Might Have Missed
- OS limitation: Only iOS 15 and iPadOS 15 (or later) users will be shown treatments
- Consistency across the App Store: Users who have seen a particular treatment, e.g., B variation, will keep seeing the same treatment (B variation) across the App Store during the whole test period. This means that the same meta treatments will be shown not only in the Search results but also on the Today, Games, and Apps tabs.
- Access limitation (spotted on the ASO Stack): It seems that PPO is only available for those who previously applied for In-App Events.
- Submission limitation (spotted on the ASO Stack): You can’t have two different submissions for review simultaneously, e.g., a new app version and a new product page treatment. The product page treatment should be submitted once the new version is approved and in the ‘ready for sale’ status.
If you submit a new app version while a test is running, the test will automatically stop.
- Limited number of tests (spotted on the ASO Stack): Only one test can be running at a given time.
- Limited Analytics (spotted on the ASO Stack): If you run a test in more than one language locale, you won’t be able to differentiate between them when it comes to reporting as there are no filters in the Acquisitions section of Analytics.
A Checklist to Set Up and Launch Your First PPO Test
- Add a reference name (up to 64 characters)
The best practice is to include the date, type of assets, and the hypotheses that are being tested (e.g., December2022_screenshots_educationalfeatures) - Select the number of treatments (up to 3)
- Choose traffic proportion (up to 75%)
- Select language locales
Note that only localizations available with the current app version can be included in A/B testing. - Estimate test duration and choose CVR improvement (optional currently not available)
Note that the test will run for 90 days or until you stop it manually within that time.
- Add treatments
Keep in mind the importance of iterative testing. Limit the number of elements changed at once so that you can easily identify what has impacted the conversion rate. Check out how to build a systematic process for ASO with the CRO Loop to improve your store listing iteratively.
Note: only screenshots and app previews are available for testing for now.
- Launch
Note that each treatment will be reviewed by the Apple team before going live. Screenshots and app previews can be tested independent of the app release, while the app’s icon should be included in the binary of your published app before running the test.
- Monitor and analyze
Keep in mind that the performance metrics will be available the day after the test begins.
You can view and analyze:
– unique impressions
– conversion rate
– improvement
– confidence level for each treatment.
The original product will be considered the baseline by default but can be changed at any time.
Final Remarks and Recommendations
As the PPO feature has just recently been officially launched, there are many things yet to be explored. Here are a few recommendations that we can share based on our first-hand experience and observations from the ASO Stack Community
- Keep in mind the existing limitations
- Submit your test for approval once your new app version is ‘ready for sale’, not simultaneously.
- Only one test can be run at a given time.
- Run one test per language locale. If you’re running a test for multiple locales, you won’t be able to differentiate them in reporting.
- Follow the iterative approach in testing
Limit the number of variables in the test to make sure you can easily identify what has brought the biggest impact to the conversion rate. Follow the CRO Loop recommendations to craft hypotheses and analyze learnings.
- Check ASO Stack updates on Slack
Join our knowledge-sharing community on Slack to get the latest updates and be part of hot-right-now discussions. And if you need specific ASO advice, feel free to reach out to us on the ASO Stack Slack Community and the Phiture team will be in touch!
If you need specific advice regarding App Store Optimization or conversion rates, feel free to reach out!
Table of Contents
