Editor’s Note: This is an opinion piece by Phiture co-founder Andy Carvell. Andy led several growth teams at SoundCloud and has consulted a large number of B2C app companies on data-driven mobile growth strategy over the past four and half years with Phiture.
If there’s one thing that will sap the life out of any data-driven growth team, it’s failing to overcome organizational resistance to experimentation. Sadly, it’s an all-too-familiar story that goes something like this:
First, the growth team starts with a ton of enthusiasm and conducts user research and data analysis to identify strong advancement opportunities. From there, the team forms hypotheses and builds a backlog of well-scoped experiments to test them. They prioritize these experiments based on some estimation of the impact and effort required (possibly with the help of a framework such as ICE or RRF).
The team-lead then excitedly briefs the leadership and other teams on the first cool experiments they are about to launch, only to have their hopes dashed when their ideas are shot down. The experiments never happen and the team is discouraged from pushing for growth in the future.
Why do experiments get shot down so often?
In an ideal world, the CEO is fully on board at the foundation of the growth team and communicates to the rest of the company that they have a remit to run experiments (responsibly) across all areas to improve metrics and accelerate organizational learning.
Typically, objections from stakeholders are more prevalent in cases where such a remit was not established top-down from the outset, thus leading individual contributors or managers in various teams to feel a responsibility to protect the product and user base from what is often perceived as well-meaning but ultimately misguided tinkering that could have disastrous consequences.
It’s entirely possible, though not always the case, that objections to experiments are based on genuine risks. Often, the objectors lack the data-centric language or methodologies to express these risks in a language that the growth lead finds convincing, so it’s important to establish a common lexicon.
It’s important to understand that this scenario always results from fear in one form or another. In the following section, I’ll break down some of the common objections and how to address the root fear behind them to avoid friction between the growth team and key stakeholders.
Fear — usually stemming from a poor understanding — of the experimental process
This is often the real reason that ideas get shut down with high regularity. People are uncomfortable and unfamiliar with what’s involved in growth experimentation. Growth experiments are often perceived as unqualified meddling in specialist subject areas (Product, Design, Marketing, Data), by generalist growth teams.
When a growth team is newly formed and needs to prove its worth within the organization, it falls on them to advance the discussion and evangelize the methodology of experimentation.
Growth is still a relatively new discipline and this needs to be understood by all growth practitioners. Part of the job of working in growth at 80% of companies today involves evangelizing the principles and process of experimentation and demystifying what is a scary topic for many. This is not a one-off task, but a continuous mission.
Fear of ‘ruining the user experience’
Damaging a well-crafted and at least somewhat performant user experience is the most commonly cited objection to growth experimentation, though this may not always fully represent the underlying concerns.
This fear tends to increase in proportion with a product’s popularity/success; stakeholders are terrified of losing the respect of their loyal user base and somehow irreparably damaging the reputation of the product with ‘growth hacks’ which will cheapen, degrade, or otherwise debase something that is working well and highly-respected. If monetization is also doing well, this fear multiplies: Why would anyone risk turning off the money tap?
Fear of ruining a great experience often stems from high emotional investment in the product that the objector has invested countless hours in designing, improving, honing. It’s ‘their baby’ and they are fiercely protective of it against what they perceive as cheap, dirty tricks or poorly thought-through changes that will degrade the user experience.
Some inherent assumptions that come in tandem with this fear include:
- Assumption: Existing, loyal users will be exposed to the experiment.
Reality: Many experiments will focus on new users. - Assumption: Many users will actively (and, perhaps, vocally) resent the experiment (i.e. they may, as a result, opt-out of communications, de-install the app, or vocally object on social media)
Reality: User reaction should be measured so that negative effects can be understood. Often, users are far less averse to the experiment than internal stakeholders. - Assumption: Skillfully-designed product changes led by a traditional feature/product team would not incur any of the above risks (ergo: only ‘designed’ changes should be permitted)
Reality: The design team should be involved from the start to ensure that experiments incorporate proper design principles and brand guidelines. However, it’s not realistic to expect that users will always love regular Product/Design updates (as opposed to growth-centric updates). The best signal is provided by the data (both quantitative and qualitative).
Fear of the experiment succeeding (Ego / Pride)
Sometimes, a key stakeholder (perhaps a long-time employee, co-founder, or C-level exec) fears a situation in which an experiment improves ‘their’ metrics, as this would make them insecure about their contribution to the business.
Nobody wants to admit that they would put their insecurities before company success, yet we are all human and subject to such irrational and emotional tendencies. If a key stakeholder or decision-maker possesses a mindset whereby growth experiments are viewed as a challenge to their considered judgment (and, by extension, their value), this often results in an impasse.
In such situations, directly challenging this stakeholder or appealing to logic/data is unlikely to result in constructive progress, since direct opposition may well be perceived as confirmation of their fear. Going ‘over the head’ of such a stakeholder and appealing to the CEO or senior leadership may be an effective course of action, but is a nuclear option.
There is no easy solution, but when it exists, it comes through genuine efforts to personally connect, empathize with another’s point of view, and to build trust and companionship outside of the context of the growth experiments at stake.
Fear of monetary loss outweighs the prospect of higher returns
Financial concerns are often raised when an app is already monetizing at a decent rate: the company becomes swiftly (and understandably) addicted to the income generated by the app and becomes deathly afraid of anything that risks restricting the flow of IAPs, advertising, or subscription revenue.
In many cases, even the prospect of generating significantly more revenue is not enough of an enticement to displace the inherent fear of killing the golden goose through ill-considered experimentation.
While financial objections are not the most common, when they occur they usually come with strong backing, often from the Chief Financial Officer (CFO). It’s important, therefore, for a successful growth lead to have a process for addressing such concerns and to take time to ensure that there is a proper understanding of the growth process and the specifics of the proposed experiment(s).
Objections of a purely financial nature are often easier to deal with than those of a fundamentally more psychological basis: Finance experts are more than comfortable with numbers, risk assessments, projections, and so forth.
In a healthy organization, the CFO and the growth lead speak the same language, albeit in different dialects, and it should be relatively straightforward to establish common ground and rules of engagement that give growth experiments a chance to bear fruit. Moreover, a CFO can often turn out to be a powerful ally to any growth practitioner; once they understand the potential upside and have a comfortable understanding of well-specified and well-instrumented experiments, they may advocate for them above more opaque and less immediately quantifiable initiatives.
Fear of legal/regulatory problems
In some organizations, the legal team has the final say on many matters. In some industries, this is extremely necessary. In others, this scenario may result from a conservative CEO or founding team. In any case, if a growth team is required to run their experiments past the legal department before they have the green light, they should be prepared to answer a lot of questions.
A good corporate lawyer has a responsibility to carefully consider anything that comes across their desk, be that a cease-and-desist, a legal summons, or a growth experiment proposal. It’s therefore reasonable and appropriate for them to ask enough questions to fully understand the situation and appraise the risks and opportunities to the company.
How to move past fear to unlock growth in your organization
Whatever the ratio within the company of ‘unqualified fear’ to ‘reasonable concern’ around growth experiments, it’s the implicit role of the growth lead to arbitrate and educate in these situations, so that diverse stakeholders coalesce around a goal and a process that everyone can get behind.
Depending on how data-driven the company is, this involves ongoing education and communication around growth processes and extreme transparency about the process followed, data collected, and analysis performed.
While most people working on digital products pay lip service to ‘test and learn’, A/B testing and using data to inform decisions, in reality, many people are deeply uncomfortable with running an experiment that would provide hard data to support or disprove a hypothesis. In many cases, there is no strong culture of using metrics within such organizations, so it’s tough to bring the conversation around to ‘looking at the numbers’ or to defining ‘success’ or ‘improvement’.
Going back to first principles, it’s fundamental for all stakeholders to agree on:
-
- The measurement criteria that will be used to evaluate performance
- How the product/feature is currently performing (using the metrics defined in step 1)
- Agree that the current version is not ‘perfect’ and that performance could be improved
- How to approach evaluating new ideas to improve performance:
– Can the stakeholders agree on some rules of engagement that facilitate the testing of new ideas, with limited audiences and under strict supervision and detailed measurement?
– If such testing is broadly agreed to be a ‘good thing’, what are the limits to this? (i.e. test group sizes, conditions for scaling a ‘winning’ experiment to more users)
– What will the growth team do to monitor for negative results? (i.e. not just ‘no uplift’ in key metrics, but looking at things like opt-outs, complaints/bad reviews, uninstalls… all of these factors can and should be measured). - Provided the rules of engagement defined in step 4 are followed by both parties, the growth team has a clear remit to run controlled experiments without seeking prior stakeholder approval. They are also duty-bound to report on success and failure, which hopefully everyone can get behind.
Making sustained efforts to nurture a culture of openness and data-informed dialogue will go a long way to establishing trust and understanding between disparate teams and stakeholders.
Table of Contents
