Editor’s note: This is a guest post from Natalie Fitch. Natalie Jane Fitch is a Senior CRM Manager of Retention and Engagement at Babbel, the highest-grossing language learning app. She is also Pun Master general of two product agile teams focused on keeping learners engaged through varied learning experiences. Before joining Babbel, Natalie was a Mobile Growth Consultant at Phiture. Natalie is a native Detroiter living in Berlin who speaks English and German and can order wine and cheese in French.
What is holding your product or brand team back from validating concepts in their backlog? Is it the time limitations of an app release calendar or a market survey? Is it limited engineering capacity to support native build out? Or an already-packed A/B test calendar? If only there were a better way…
When I first joined Babbel, the language learning app, I wasn’t in a traditional CRM role. I was on a task force called Help Me Learn. Our mandate was to validate that creative communications and experiences outside of traditional learning content could impact learner engagement.
We knew a couple of things about our product. Firstly, we have high-quality, engaging language learning content. We had an impressive subscriber engagement rate and scores of experienced language experts on board to prove it. Secondly, our product works. We had dozens of case studies and several academic research projects which empirically validated Babbel’s efficacy.
What we didn’t know was how experiences outside of this content would or could affect engagement with the content itself. Could we offer our learners a way to get inspired and motivated to learn a new language? Could we build a product that guides them through their language learning journey, from Bonjour to their first trip to Paris? Using only CRM and social media, Help Me Learn set out to validate these hypotheses.
Not only were we able to validate that a higher focus on engaging our learners outside of learning content would lead them to dedicate more time to achieving their goals, we validated that CRM is a channel in which feature development, feature messaging, and brand messaging can be tested and validated without the time and effort limitations of traditional testing methods.
Feature Development
One of the first hypotheses that we set out to validate was that giving learners a sense of progress towards a goal would lead to higher engagement with learning content. We set up a series of campaigns, one of which congratulated learners on completing a certain proportion of a course (we focused on 33% and 67%). We conducted manual tests which involved downloading cohorts which met the send criteria from our analytics tool and uploading them into our CRM tool daily. After a couple of weeks, we were able to validate that nudging learners back to learning by presenting their current progress in a course led to a 45% uplift in learning in the following 7 days over the control.
Without the engineering effort and time necessary to build this feature out natively and deploy an A/B test in-app release cycles on both platforms, we were able to validate that giving learners a sense of progress towards a goal would lead to higher engagement with learning content. Our manual CRM tests not only shaped the product roadmap but also contributed to the decision to dedicate an entire product team to helping learners feel success and progress in their learning journey.
Feature Messaging
Our testing doesn’t stop at feature development. Another form of testing that we often do is feature positioning. What messaging best piques our learners’ interest in new features and content?
Ahead of launching a referral feature within our product, for example, we ran a creative test in CRM to validate which type of messaging leads to higher rates of referral. We simply added a content block to the bottom of our learning summary email, which sends weekly to our most engaged learners. We tested three versions of the email against each other for a week and watched which email saw the most conversion.
For the referral feature’s MVP launch within the product, product teams were able to avoid a natively built-out A/B/C test which would rely on the time constraints of an app release schedule to validate the messaging. Instead, they were able to use the learnings from the quick CRM test to implement the creative copy variant that was proven to be most successful.
Brand Messaging
CRM testing doesn’t stop with the product; we also work closely with brand teams to validate new tone-of-voice and visual identity concepts. Brand discovery work can often be a tedious and long process. Several channels often work in parallel to answer questions about where the brand can and should go. CRM is no exception.
The direct line of communication to existing learners (i.e., leads who have already converted) is what makes CRM a particularly valuable resource for brand testing. By informing inbound marketing channels of insights from existing subscribers, we can focus our lead attraction efforts on the leads who are most likely to convert and later engage.
We recently ran a brand visual test in one of our newsletters to do exactly this. We split our group in two and showed half of them a photograph of a person happily on their phone on the go. In the other, an illustration of a person on the go on their phone.
With such a large sendout, results were gathered almost instantaneously, and within a week, we were able to definitively recommend to our Insights & Brand team that they move forward with illustrated images as opposed to photographs in other channels.
The Help Me Learn taskforce has long since disbanded and we have in the meantime built out a team of talented and driven CRM Managers to own and develop the subscriber lifecycle in the CRM channels. Though our mandate as a team has changed, we still set aside bandwidth and creative energy for product and brand testing in our channel. Our strong relationship with product and brand teams has led to the CRM team being considered the first line of validation for teams who are batting around product, messaging and visual concepts.
Table of Contents
