# iOS15 and Testing: A Strategy for Conversion Rates

- Link: https://phiture.com/asostack/ios15-and-testing-a-strategy-for-conversion-rates/
- Published: 2021-09-28T11:09:17+01:00
- Author: Joy Staples and Winda Adjani

![](https://phiture.com/wp-content/uploads/2021/09/Frame-104-8-1024x544.png)

Now that iOS 15 is here, marketers everywhere will soon be facing the new upcoming
features with the same old question: how can I use these to increase conversion?
With the new iOS 15 changes, the chance to use ASO to reach and convert more users
is bigger than ever, and this is a train everyone wants to board. Once the new product
page optimization (PPO) feature rolls out, you’ll need to rethink your strategy 
in order to use the new capabilities to your advantage, and not just reuse the same
strategy you were (or weren’t) using for Google Play.

To recap: the new iOS 15 feature will allow you to test **icons, screenshots and
app previews**, enabling you to find the best variants (aka, new iterations of your
assets) for each market, and to gather data about each market’s preferences. This,
of course, will be done by testing each one of the app’s assets, raising the questions:**
how should I test, what metrics exactly should I look at when testing? And how should
those be interpreted?** In this article, we’ll answer these questions, looking at
both the metrics you should track as well as the best practices regarding the actual
testing. If you’d like to look more closely at the upcoming iOS 15 changes, feel
free to check out the previous article we published around it, on [getting your UA/ASO strategy ready](https://phiture.com/asostack/ios-15-get-your-ua-aso-strategy-ready-for-product-page-optimization/).

So, let’s get started.

Table of Contents

 * [The Metrics](https://phiture.com/asostack/ios15-and-testing-a-strategy-for-conversion-rates/#the_metrics)
 * [Best strategies and practices](https://phiture.com/asostack/ios15-and-testing-a-strategy-for-conversion-rates/#best_strategies_and_practices)
 * [Conclusion:](https://phiture.com/asostack/ios15-and-testing-a-strategy-for-conversion-rates/#conclusion)

## **The Metrics**

Let’s say you finished your first testing cycle, and now you have your results. 
Looking at all the different metrics in the analytics, which ones should you focus
on? And why? Well, regarding the metrics we can divide things into two groups: the**
primary metrics and the thresholds**. As the name explains, the primary metrics 
are the most important ones, the ones that help you measure success (what moves 
the needle). The thresholds are the numbers you should be striving for in order 
to get valid results. As for iOS 15 and testing, let’s have a look at these two 
groups:

**Primary metrics:**

We have a few big ones we should focus on:

**Improvement:** According to Apple, developers will be able to see the improvement
of the testing variant compared to the control variant. It’s not clear whether Apple
will show the changes in conversion rate or the actual downloads, but the improvement
is the primary signal for developers to understand how good (or bad) the testing
variant is.

**Downloads** **(App Units on iOS)**: Downloads are the core metric to evaluate 
the magnitude of a win (or a loss). Any work you do in PPO is to increase the likelihood
of users downloading your app once they see it, as well as get more users to download
the app.

**Conversion rate (CVR) between control and test variants: **This is always one 
of the most important metrics to measure, as it allows us to see which variant is
performing better and bringing in higher conversion. That’s also the improvement
Apple will likely show developers. When it comes to picking the right variant to
go with, this is the main thing to consider, and the holy grail of metrics for any
marketer. One note, though: don’t just look at the download-level conversion rate.
Once you apply the winner, you should also monitor the installs and in-app events
in order to understand whether or not the new creatives bring you the right audience.

**Minimum thresholds:**

In order to make sure the experiment results are valid, your metrics should reach
the following minimum thresholds. 

**Days running for test**: Run for at least a full business cycle (seven days). 
If you want to collect more data, run it for even longer than that – after all, **
the App Store allows you to run experiments up to 90 days**. It’s important to be
aware of anything else that might be going on during this time that can affect your
CVR. For example other marketing campaigns, the fact that you are being featured
on Apple, or any outside factors (e.g. events, protests, etc). All of these factors
may affect the experiment results, which is why a minimum of seven days is recommended.
After all, you want a representative sample, not a convenient one.

**Impressions: **How many users are actually seeing your app? The more the better,
of course, but it’s not quite that simple. In order to make sure you’re always using
a viable sample, It’s important to have a minimum number of installs per variant–
depending on the size of your app. The sample size should be based on your current
conversion rate and the expected uplift. There are a lot of sample size calculators
online. Use one of those and make sure you only apply a winning variant after reaching
the predefined sample size.

**Statistical significance:** The metrics will be run through a Bayesian model, 
meaning that Apple might show developers how statistically significant the result
is, just like Google does. Google calls out winners when the confidence level reaches
90%. Hopefully, Apple will provide a higher confidence level (95% or 99%) for more
accurate results. 

## **Best strategies and practices**

Regarding the actual experimenting and testing as well as the results, there are
a few things you should keep in mind. The following best practices/strategies can
help you find the right way in which to best conduct and organize your experiment
period, and also how to interpret the results properly.

Let’s start with best practices regarding testing:

**Need to create a testing strategy and a testing cycle? Don’t worry, the CRO Loop
is here to help.
 
Our Conversion Rate Optimization Loop allows you to create the
ideal testing strategy for CVR growth, step by step, and with a full cycle. You 
can have a look at the framework below, and read more about it in our [dedicated article](https://phiture.com/asostack/building-a-systematic-process-for-aso-with-the-cro-loop-d1c4bc7ca725/).
Having a full testing cycle is essential, as it is to keep the following points 
in mind:

![](https://phiture.com/wp-content/uploads/2021/09/image2-5-1024x575.png)

Our CRO Loop

 

**Iterative testing**
 This is a big one. Make sure not to make multiple, big changes
at the same time, so that when one variant wins there’s no doubt as to what made
it successful. For example: in a screenshots design test, focus just on changing
the font and nothing else. What that means is having a variant A and a variant B
with different fonts. Whether variant A or variant B wins, you’ll know the reason
why.

**Reviews research:
 
Read through your app reviews (and those of your competitors!)
to see what it is that users like most about it (sometimes what we think is the 
best part of the app isn’t the same thing that users think). Based on this, you 
can try to see what people are searching for, and then you can highlight these features
better.

**Localize:**

 * Keep in mind: There are 7.8 billion people in the world, and only 1.35 billion
   of them speak English – meaning, never underestimate the power of localization!
 * Localizing content has proven to be effective in increasing CVR. The assets you
   can localize are: copy, in-app images, and creatives. Stick to the best practices
   from each particular country. 
 * Once you’ve done your A/B testing and have a clear winner, perform a thorough
   research on your app’s locales and adapt your screenshots accordingly. The in-
   app images that work in the US, for example, may be totally uninteresting to 
   the users in the Asia-Pacific (APAC) region. It’s always useful to gather learnings
   from the locale with the biggest user base and apply those learnings to other
   locales, but it’s also important not to assume that you will have the same successes
   all over the world. 

Now, let’s look at the results and how to actually measure and report on your experiments:

 * Main metrics: **App Store: App Units
    
   App units are first time downloads, which,
   as previously discussed, are essential in finding out how fast (or slowly) your
   audience is growing.
 * Regardless of how iOS 15 displays the results/wins/losses (or “improvement”, 
   as Apple calls it), you should focus on the uplift the variant brings. Since 
   Apple’s new update isn’t out yet, we’ll base this on Google Play A/B testing.
   Here’s what you should focus on:
    - How much uplift percentage did the variant bring?
    - How many more installs that specific variant will bring you in a year when
      compared to the control variant? We call this annualized install uplift.

After research, it’s time to hypothesise for new potential experiments – turn each
idea into a hypothesis (strategy) – and later, once the experiments have finished
running, confirm/disprove the hypothesis and turn it into learning.

Whichever experiments didn’t win, take them as a learning and not as a  “waste”.
Move on from bad ideas – like we’ve written about before, in an old but still relevant
article, [kill an idea = stimulus for learning](https://phiture.com/mobilegrowthstack/scale-iterate-or-kill-f4a8e8987d4e/).
And if it works, scale it.

![](https://phiture.com/wp-content/uploads/2021/09/image1-6.png)

How to work with experiments

## **Conclusion:**

You could A/B test the A/B testing itself, since there are so many different ways
of doing it. It’s important to have a strategy in mind, along with best practices,
as well as to know what to look for in your results and how to interpret them. By
following these best practices, you’ll be able to properly test and, just as important,
interpret the testing results with the aim of increasing your CVR. With the new 
iOS 15 features, it’s never been more important to be on top of your game – and 
we’re here to help you do just that.

As always, feel free to reach out with any questions.