# What Kills Data-Driven Growth Teams?

- Link: https://phiture.com/mobilegrowthstack/what-kills-data-driven-growth-teams/
- Published: 2021-04-14T08:33:55+01:00
- Author: Andy Carvell

_⌊What Kills Data-Driven Growth Teams? by Andy Carvell⌉⌊What Kills Data-Driven Growth
Teams? by Andy Carvell⌉Editor’s Note: This is an opinion piece by Phiture co-founder
Andy Carvell. Andy led several growth teams at SoundCloud and has consulted a large
number of B2C app companies on data-driven mobile growth strategy over the past 
four and half years with Phiture._

If there’s one thing that will sap the life out of any data-driven growth team, 
it’s failing to overcome organizational resistance to experimentation. Sadly, it’s
an all-too-familiar story that goes something like this:

First, the growth team starts with a ton of enthusiasm and conducts user research
and data analysis to identify strong advancement opportunities. From there, the 
team forms hypotheses and builds a backlog of well-scoped experiments to test them.
They prioritize these experiments based on some estimation of the impact and effort
required (possibly with the help of a framework such as [ICE](https://tech.trello.com/ice-scoring/)
or [RRF](https://phiture.com/mobilegrowthstack/rrf-a-framework-for-building-impactful-notifications-73c7b91c45a7/)).

The team-lead then excitedly briefs the leadership and other teams on the first 
cool experiments they are about to launch, only to have their hopes dashed when 
their ideas are shot down. The experiments never happen and the team is discouraged
from pushing for growth in the future.

Table of Contents

 * [Why do experiments get shot down so often?](https://phiture.com/mobilegrowthstack/what-kills-data-driven-growth-teams/#why_do_experiments_get_shot_down_so_often)
 * [Fear — usually stemming from a poor understanding — of the experimental process](https://phiture.com/mobilegrowthstack/what-kills-data-driven-growth-teams/#fear_%e2%80%94_usually_stemming_from_a_poor_understanding_%e2%80%94_of_the_experimental_process)
 * [Fear of ‘ruining the user experience’ ](https://phiture.com/mobilegrowthstack/what-kills-data-driven-growth-teams/#fear_of_%e2%80%98ruining_the_user_experience)
 * [Fear of the experiment succeeding (Ego / Pride)](https://phiture.com/mobilegrowthstack/what-kills-data-driven-growth-teams/#fear_of_the_experiment_succeeding_ego_pride)
 * [Fear of monetary loss outweighs the prospect of higher returns](https://phiture.com/mobilegrowthstack/what-kills-data-driven-growth-teams/#fear_of_monetary_loss_outweighs_the_prospect_of_higher_returns)
 * [Fear of legal/regulatory problems](https://phiture.com/mobilegrowthstack/what-kills-data-driven-growth-teams/#fear_of_legalregulatory_problems)
 * [How to move past fear to unlock growth in your organization](https://phiture.com/mobilegrowthstack/what-kills-data-driven-growth-teams/#how_to_move_past_fear_to_unlock_growth_in_your_organization)

## **Why do experiments get shot down so often?**

In an ideal world, the CEO is fully on board at the foundation of the growth team
and communicates to the rest of the company that they have a remit to run experiments(
responsibly) across all areas to improve metrics and accelerate organizational learning.

Typically, objections from stakeholders are more prevalent in cases where such a
remit was not established top-down from the outset, thus leading individual contributors
or managers in various teams to feel a responsibility to protect the product and
user base from what is often perceived as well-meaning but ultimately misguided 
tinkering that could have disastrous consequences.

It’s entirely possible, though not always the case, that objections to experiments
are based on genuine risks. Often, the objectors lack the data-centric language 
or methodologies to express these risks in a language that the growth lead finds
convincing, so it’s important to establish a common lexicon. 

It’s important to understand that this scenario always results from **_fear_** in
one form or another. In the following section, I’ll break down some of the common
objections and how to address the root fear behind them to avoid friction between
the growth team and key stakeholders.

## **Fear — usually stemming from a poor understanding — of the experimental process**

This is often the real reason that ideas get shut down with high regularity. People
are uncomfortable and unfamiliar with what’s involved in [growth experimentation](https://phiture.com/mobilegrowthstack/scale-iterate-or-kill-f4a8e8987d4e/).
Growth experiments are often perceived as unqualified meddling in specialist subject
areas (Product, Design, Marketing, Data), by generalist growth teams.

When a growth team is newly formed and needs to prove its worth within the organization,
it falls on them to advance the discussion and evangelize the methodology of experimentation.

Growth is still a relatively new discipline and this needs to be understood by all
growth practitioners. Part of the job of working in growth at 80% of companies today
involves evangelizing the principles and process of experimentation and demystifying
what is a scary topic for many.  This is not a one-off task, but a continuous mission.

## **Fear of ‘ruining the user experience’ **

Damaging a well-crafted and at least somewhat performant user experience is the 
most commonly cited objection to growth experimentation, though this may not always
fully represent the underlying concerns.

This fear tends to increase in proportion with a product’s popularity/success; stakeholders
are terrified of losing the respect of their loyal user base and somehow irreparably
damaging the reputation of the product with ‘growth hacks’ which will cheapen, degrade,
or otherwise debase something that is working well and highly-respected. If monetization
is also doing well, this fear multiplies: Why would anyone risk turning off the 
money tap?

Fear of ruining a great experience often stems from high emotional investment in
the product that the objector has invested countless hours in designing, improving,
honing. It’s ‘their baby’ and they are fiercely protective of it against what they
perceive as cheap, dirty tricks or poorly thought-through changes that will degrade
the user experience.

Some inherent assumptions that come in tandem with this fear include:

 * _Assumption:_ Existing, loyal users will be exposed to the experiment.
    _Reality:_
   Many experiments will focus on _new users._
 * _Assumption:_ Many users will actively (and, perhaps, vocally) resent the experiment(
   i.e. they may, as a result, opt-out of communications, de-install the app, or
   vocally object on social media)
    _Reality:_ User reaction should be measured 
   so that negative effects can be understood. Often, users are far less averse 
   to the experiment than internal stakeholders.
 * _Assumption:_ Skillfully-designed product changes led by a traditional feature/
   product team would not incur any of the above risks (ergo: only ‘designed’ changes
   should be permitted)
    _Reality:_ The design team should be involved from the 
   start to ensure that experiments incorporate proper design principles and brand
   guidelines. However, it’s not realistic to expect that users will always love
   regular Product/Design updates (as opposed to growth-centric updates). The best
   signal is provided by the data (both quantitative and qualitative).

## **Fear of the experiment succeeding (Ego / Pride)**

Sometimes, a key stakeholder (perhaps a long-time employee, co-founder, or C-level
exec) fears a situation in which an experiment _improves_ ‘their’ metrics, as this
would make them insecure about their contribution to the business.

Nobody wants to admit that they would put their insecurities before company success,
yet we are all human and subject to such irrational and emotional tendencies. If
a key stakeholder or decision-maker possesses a mindset whereby growth experiments
are viewed as a challenge to their considered judgment (and, by extension, their
value), this often results in an impasse.

In such situations, directly challenging this stakeholder or appealing to logic/
data is unlikely to result in constructive progress, since direct opposition may
well be perceived as confirmation of their fear. Going ‘over the head’ of such a
stakeholder and appealing to the CEO or senior leadership may be an effective course
of action, but is a nuclear option.

There is no easy solution, but when it exists, it comes through genuine efforts 
to personally connect, empathize with another’s point of view, and to build trust
and companionship outside of the context of the growth experiments at stake.

## **Fear of monetary loss outweighs the prospect of higher returns**

Financial concerns are often raised when an app is already monetizing at a decent
rate: the company becomes swiftly (and understandably) addicted to the income generated
by the app and becomes deathly afraid of anything that risks restricting the flow
of IAPs, advertising, or subscription revenue.

In many cases, even the prospect of generating significantly more revenue is not
enough of an enticement to displace the inherent fear of killing the golden goose
through ill-considered experimentation.

While financial objections are not the most common, when they occur they usually
come with strong backing, often from the [Chief Financial Officer](https://www.investopedia.com/terms/c/cfo.asp)(
CFO). It’s important, therefore, for a successful growth lead to have a process 
for addressing such concerns and to take time to ensure that there is a proper understanding
of the growth process and the specifics of the proposed experiment(s).

Objections of a purely financial nature are often easier to deal with than those
of a fundamentally more psychological basis: Finance experts are more than comfortable
with numbers, risk assessments, projections, and so forth.

In a healthy organization, the CFO and the growth lead speak the same language, 
albeit in different dialects, and it should be relatively straightforward to establish
common ground and rules of engagement that give growth experiments a chance to bear
fruit. Moreover, a CFO can often turn out to be a powerful ally to any growth practitioner;
once they understand the potential upside and have a comfortable understanding of
well-specified and well-instrumented experiments, they may advocate for them above
more opaque and less immediately quantifiable initiatives.

## **Fear of legal/regulatory problems**

In some organizations, the legal team has the final say on many matters. In some
industries, this is extremely necessary. In others, this scenario may result from
a conservative CEO or founding team. In any case, if a growth team is required to
run their experiments past the legal department before they have the green light,
they should be prepared to answer a lot of questions.

A good corporate lawyer has a responsibility to carefully consider anything that
comes across their desk, be that a cease-and-desist, a legal summons, or a growth
experiment proposal. It’s therefore reasonable and appropriate for them to ask enough
questions to fully understand the situation and appraise the risks and opportunities
to the company.

## **How to move past fear to unlock growth in your organization**

Whatever the ratio within the company of ‘unqualified fear’ to ‘reasonable concern’
around growth experiments, it’s the implicit role of the growth lead to arbitrate
and educate in these situations, so that diverse stakeholders coalesce around a 
goal and a process that everyone can get behind.

Depending on how data-driven the company is, this involves ongoing education and
communication around growth processes and extreme transparency about the process
followed, data collected, and analysis performed.

While most people working on digital products pay lip service to ‘test and learn’,
[A/B testing](https://phiture.com/mobilegrowthstack/the-a-b-testing-framework-how-to-level-up-your-a-b-experimentation/)
and using data to inform decisions, in reality, many people are deeply uncomfortable
with running an experiment that would provide hard data to support or disprove a
hypothesis. In many cases, there is no strong culture of using metrics within such
organizations, so it’s tough to bring the conversation around to ‘looking at the
numbers’ or to defining ‘success’ or ‘improvement’. 

Going back to first principles, it’s fundamental for all stakeholders to agree on:

    1. The measurement criteria that will be used to evaluate performance
    2. How the product/feature is currently performing (using the metrics defined in
       step 1)
    3. Agree that the current version is not ‘perfect’ and that performance could be
       improved
    4. How to approach evaluating new ideas to improve performance:
        – Can the stakeholders
       agree on some rules of engagement that facilitate the testing of new ideas, 
       with limited audiences and under strict supervision and detailed measurement?–
       If such testing is broadly agreed to be a ‘good thing’, what are the limits 
       to this? (i.e. test group sizes, conditions for scaling a ‘winning’ experiment
       to more users) – What will the growth team do to monitor for negative results?(
       i.e. not just ‘no uplift’ in key metrics, but looking at things like opt-outs,
       complaints/bad reviews, uninstalls… all of these factors can and should be measured).
    5. Provided the rules of engagement defined in step 4 are followed by both parties,
       the growth team has a clear remit to run controlled experiments without seeking
       prior stakeholder approval. They are also duty-bound to report on success and
       failure, which hopefully everyone can get behind.

Making sustained efforts to nurture a culture of openness and data-informed dialogue
will go a long way to establishing trust and understanding between disparate teams
and stakeholders.