At a talk aimed mostly at growth teams, we opened by asking the room for a quick show of hands: who here works on a creative team or has a creative role? Only a couple of hands went up, with some hesitation, which fit a presentation built mainly for growth marketers, though we noted there was plenty in it for creative professionals too.
I’ve spent 13 years as a creative digital marketer. I originally studied biochemistry and math in school before switching to writing, so I think of myself as a nerd in addition to a creative. I like to illustrate that combination with a picture of Fibonacci, another math nerd who loved beauty: Fibonacci’s spiral is often placed over famous artworks, like Vermeer’s Girl with a Pearl Earring and the Mona Lisa, as a supposed ideal ratio, though it never quite lines up perfectly. I also hold a US patent for a system that analyzes the performance of UA creatives and then iterates on them using machine learning, so I’ve spent a long time working from both sides of the data and creative divide.
One of the first questions growth leaders ask me is how to improve the data literacy on their creative team. My answer starts by pointing out that math comes in orders of magnitude of difficulty. The most difficult is chaos theory. Second is UA campaigns and campaign data. After that comes dividing the bill at dinner. Then come ASO test results, which aren’t particularly complicated: once a test reaches statistical significance, the platform gives you a red bar and a green bar, and if the green bar is larger, you can generally assume it did better than the control.
If the data isn’t complicated for ASO test results, that means the real difficulty lies somewhere else in the process.
Table of Contents
Trust the platform, with some nuance
I said this to someone earlier the same day, and she pushed back: you can’t actually trust the platforms; they lie to you all the time. That deserved a more nuanced answer. Yes, you look back at your tests and adjust for noise, and you make sure you have enough volume over a certain amount of time. But when it comes to individual tests, if you trust Apple and Google, you can trust the red and green bars.
UA campaigns are a different story; it’s harder to make sure every creative is getting the same scale, and even when it’s possible, it’s always more costly.
The real bottleneck is ecosystem literacy, not statistics
I’ve been consistently shocked, over the years, as the industry has grown and the tools have become more user-friendly, that very few companies take the time to educate their creatives about what the actual digital ecosystem for ads or ASO looks like. I’ve worked with designers where we would mention a specific ad network, or talk about DSPs, and a year into working there, they would ask me what a DSP actually was. No one had ever taken the time to explain it.
Most of the heavy lifting that has to happen, and most of the education that needs to happen, is logistical, and it needs to happen before creatives ever go live. Creative teams need to understand the ecosystem: the many different ways a company is doing its advertising and marketing, and all of the logistical concerns that come with that. I think it’s important to walk creative teams through the whole marketing funnel and to train them up as if they were growth marketers, so they understand where their creatives are actually being seen. That one step does about half the work of explaining results to a creative team later, because once people understand the ins and outs, or even just the gist, of different ad networks, it’s rare that anyone has a straightforward answer to “what’s your best ad,” and understanding the ecosystem is what makes that answer make sense.
For organizations that work on only one or two platforms, I spend time teaching the algorithm specifically. On Meta, for example, I teach creative teams about ad groups and how the platform optimizes groups of creatives, because otherwise teams start to think small: what’s the point of trying all these different versions if half of them never get seen? The answer is that the algorithm is going to pick a champion, and you have to deal with that unless you’re willing to force spend.
Map creative tactics to the metrics they move
The next most valuable thing, in my view, is to map creative to metrics. If a company is struggling with a particular part of the funnel, there are different creative tactics suited to that problem. This mapping is loose, but it gives teams a place to start.
| Funnel stage | Creative lever | Metric to watch |
|---|---|---|
| Awareness | Design, the visual itself | Views, impressions |
| Interest | Messaging, the hook that captures attention | CTR |
| Consideration | The content of the message, separate from how it’s delivered | Installs, CTI |
| Conversion | Integrity of the full creative experience, not one isolated asset | Conversion rate |
If a team is struggling to get views, or has an awareness problem, design is the first thing to tackle, since it’s what most people respond to in a visual ad. Messaging is where interest is captured, and that loosely ties to clicks and CTR. Consideration is the content of the message, which we separate from messaging itself: what you’re saying is different from how you’re saying it, and that correlates to installs and CTI. Conversion is where things get more complicated, because creatives aren’t as straightforward by that point, and it comes down to the integrity of the whole funnel and everything the user sees along the way.
Use the scientific method, and isolate one variable
I teach the scientific method to a lot of the creative teams I work with, because I think understanding logic matters more than understanding statistics when it comes to getting creatives to work with data. Since ASO data, especially, can be very simple, the real responsibility sits with the logic used to design the creative, so that whatever hypothesis is being tested, the results actually hold up.
That starts with being clear about what kind of test you’re running. I like to use movie posters as an example, because studios change the design a lot between releases. I point to the Pete’s Dragon remake poster against the one from the original film from the ’80s: the content is the same, but the design is completely different. That’s a design test. Nothing about the messaging or the value proposition changes, and the audience isn’t different either. You’re only trying to capture attention in a different way. Being this specific about what you’re testing is what helps a creative team interpret the results in a way that makes sense, because the point of any creative test is to decide what to do next, not just to find a performant creative.
A messaging test looks different, and my favorite example is Campbell’s Soup. Years ago, while building the machine-learning ad-analysis platform tied to my patent, my team made Campbell’s Soup ads to test it. The theory was that you can have as many heartwarming family scenes as you want, but not everyone likes their family, so you can’t use family imagery to sell to every individual person. The content and the value proposition stayed the same; you’ll want to drink soup because of your home life, but the messaging changed. Again, that’s isolating the independent variable and making sure that whatever data comes back can lead to an actionable decision.
None of this involves statistical analysis or any math at all. It’s about making sure the creative team has done everything it can before a creative goes live, to make sure the results will actually be useful.
Don’t engineer creative expertise out of the test
This part is a little trickier to talk about, because it’s hard to oversell the importance of creative expertise. I often work with clients who want very rigid, laboratory-safe tests, whether in the App Store or in UA campaigns, and as a result they start thinking smaller and smaller, trying to limit their variables as much as possible so that when they see a half a percent difference in IPM, they can point to exactly why it happened. Taking a broader view of creative and the creative process means you can analyze creatives that are pretty wildly different from each other and still get actionable results.
I use two posters for Evil Dead 2 to make this point: one released in the United States, one in Japan. There isn’t much that’s similar between them. They’re both marketing the same movie, but the design is wildly different, the tone is wildly different, and even how the main characters are portrayed is very different. Taking a broader view of how creative gets made, you can think of this less as a test of one specific execution and more as a test of whether you’re using local culture experts in your marketing funnel if you’re advertising in a lot of different locations. You could, in theory, translate the Japanese poster into English and the American one into Japanese and run an A/B test between them, but that isn’t really how movie posters work. It’s more worth highlighting as a test of how the creative gets produced in the first place.
A more specific version of this came up in my first job at an agency, where every new client’s first question was what the perfect button color should be. It drove me a little crazy. So I ran a fairly large test: ten ads, three button variations on each. One used a red button, because repeated past testing had shown red generally performed best. One had a button the designers made specifically for that ad, with no guidelines beyond “make a button that fits this ad.” One ad ran with no button at all. Every single time, the button designed by the designer for that specific ad won.
I think that’s a good way to show that creative expertise isn’t something you should engineer out of your creative tests. I know it can sound a little self-serving coming from a creative marketer, but these kinds of tests hold up again and again.
Report business-relevant deltas, not raw metrics
When it comes to UA campaigns specifically, there’s more nuance than with ASO, and it’s harder, or at least more costly, to get every creative the same scale. So the question becomes how to show a creative team which ad is actually doing best, or how to build a dashboard or report that makes that clear.
My approach is to talk about almost every metric as a delta over the average performance of that campaign, rather than comparing raw CTR, CTI, or IPM numbers side by side. That creates a dynamic benchmark for whatever time period and network you’re looking at, and from there you set a threshold above which something counts as successful; in my example, 150% over the average.
| Example ad | Delta vs. average IPM | Status |
|---|---|---|
| Creative A | +210% | Above 150% threshold |
| Creative B | +40% | Below threshold |
| Creative C | −15% | Underperforming |
Looking at the creative production pipeline as part of the creative process also means bringing in relevant business metrics. I worked with a client whose creative team was split roughly in half between 2D designers and video designers, for a children’s education app. The video team kept getting despondent because their ads were consistently showing up below the 2D ads in reports, underperforming on raw installs and subscriptions. Once I started showing them cost per paying install and cost per subscription by platform, something the growth team hadn’t done before I got there, the two ad types turned out to be comparable. Creative teams aren’t always brought into this kind of data, since they’re usually assembled around a specific production expertise, and they benefit a great deal from seeing it.
The 150% benchmark itself gets reverse-engineered from the team’s own capacity. If a creative team can produce a new video every two weeks, and the account loses roughly 30% of creative performance every two weeks on its main networks, then hitting 150% of the average is what it takes to keep introducing ads that stay ahead of that decay. This gives a creative team something to work toward that isn’t just a feeling, and I think it’s good not only for the business and the performance of the creatives, but for morale. It’s nice for a design team to have targets to hit, like the rest of the marketing organization, and to be able to point to something on paper that shows they’re achieving them.
Build a real creative testing roadmap
When creative teams are thinking through new creative, I think it’s important to keep the scientific method built into everything. My team uses brainstorm boards where current screenshots sit at the top, so every iteration and every test gets discussed as an iteration against a control. That keeps the hypothesis and the testing mindset in view, and it lets a creative team start thinking of every test as part of a larger decision tree, rather than taking stabs in the dark.
Without that structure, I like to compare what happens to “like treats like,” an idea from medical history associated with Pliny the Elder, an ancient Roman physician. Pliny believed things like crushed bees could treat stinging pain, or that onions could treat colds, runny noses, and irritated eyes, on the logic that whatever caused a symptom must be the key to solving it. A broken creative pipeline that isn’t using data to inform itself can end up working the same way. When an ad gets too prescriptive- download game, buy coin, feel good about it- and a creative team isn’t making its own decisions, ingesting data, and acting on it, the ads can end up feeling flat, especially if hypotheses get narrowed down just to produce actionable data without keeping the bigger picture in mind.
I lay out three stages of advancement for a creative team.
| Stage | What the creative team does |
|---|---|
| Logical | Helps form hypotheses, sits in on strategy conversations, assists with research (give them the same research tools, like an AppTweak-style license, that the rest of the team already has) |
| Strategic | Builds its own testing roadmap, separate from the ASO or UA roadmap, usually starting with evergreen creative so it doesn’t disrupt scheduled launches |
| Driver | Manages its own testing campaigns, including a slice of real ad spend |
That third stage, handing some amount of money to the creative team to run its own testing campaigns, makes some people nervous. My view is that the best creative teams I’ve been part of have been behind the wheel on at least some creative spend. That doesn’t mean handing over login credentials and walking away, but if the creative team is actually launching its own creatives, even without help, it’s a strong learning experience, and it cuts down the amount of information that has to be driven by someone else. No one likes endless meetings or endless documentation, and an actively involved creative team means a lot less of both.
The takeaway
The core point I want to make is that none of this is really about math. ASO data is simple once a test reaches significance. What takes work is everything around it: making sure creative teams understand the ecosystem they’re designing for, mapping creative tactics to the metrics they actually move, running tests that isolate one variable at a time, protecting creative expertise instead of testing it out of existence, translating results into business-relevant numbers, and building a real testing roadmap that gives creative teams more ownership over time.
For growth leaders, that suggests a different starting point than asking how to improve a creative team’s data literacy. It’s worth asking instead how much of the ecosystem, and how much of the business logic behind the numbers, the creative team has actually been given. My experience, across 13 years and a long run of these tests, is that creative teams given this kind of structure don’t just produce better results. They become part of deciding what to test next.
