Getting into the minds of your users is one of the most challenging tasks for a mobile product manager. In this interview with Mobile Growth Stack, Adeline Lee, Director of Growth at Framer, previously at Clue, shares her experience with user research for apps from a product perspective.
With all the wealth of information available from analytics and other sources — why do apps need user research?
It is helpful to always be in touch with the thoughts of your users. This should be something teams do on an ongoing basis with their existing users as needs and behavior changes over time.
What is true 18 months ago could be different today as trends, other influencing factors, and new features in the app might influence the way users perceive and use the app.
When the product and design team is completely immersed in the current service offering, it is quite easy to tunnel vision on ideas.
Existing users might be similarly engaged with the service and that makes it hard to see what improvements they might find useful or helpful. Hence, talking to potential users (users that might be using a similar service or have an interest to the topic) might shed some insights into their thoughts on specific topics which can lead to discovering new opportunities that the team didn’t consider. Since they aren’t users yet, they are not primed to provide answers the interviewers expect but a more blunt and honest response.
The third group of users within user research are non-users; people that have never used a similar service or people who are skeptical. Talking to this group of people is interesting because you want to figure out what are their barriers to consider using such a service or app. Learning from them can be used to design features and services which lower the barrier to become a user. User research typically covers these user groups (existing, potential, and non-users) and a good overall picture on product development can be inferred from there.
Where do you see user research within the organization?
User research needs to be an integral part of product development. If the hypothesis is strong, then user research will provide plenty of ideas on how to implement the findings. Usually, user research is done by a product manager and a designer, so they can work directly with the results and implement them into the product.
How do you convince other teams to participate in user research?
From experience, once a team member attended a user research session, they see how the app or feature is used from the perspective of users. It’s very helpful especially for team members who were initially skeptical about product roadmaps which were formed through user research results. User research helps here with the process of prioritizing ideas, especially for the ones which might sound very advanced and sexy but actually do not appeal to users.
How do you approach people to interview?
The easiest group of users to find is existing users. Normally through email and notifications, we ask users if they’re interested to help us improve the app. Then we get them to sign up through a form that consists of a series of questions, such as location, device, and topics related to the usage of the app to segment them into specific groups. We can use this information to filter for the types of interviews we want to conduct.
Sample email we send out to Clue users for user testing recruitment
For potential users, going to a co-working space or local hangout spot would be the simplest approach, since a majority of people there would be digital natives and are probably open to spare some time to speak to us.
To get the non-users, similar techniques like the above apply. If we want the group to meet narrow criteria, we hire an agency to find them for us. Those criteria can be very specific, like types of contraceptives, diseases, or sexual orientation.
Do you use individual or group interviews?
We only do 1-on-1 interviews because the topic of the cycle is very personal and unique. With groups, it is easy to get an incorrect impression of the topic if there is a dominating character in the room.
Do you see big differences between these three groups (existing users, potential users, and non-users)?
Fundamentally, each of these groups is distinctively different. Prior to any interview, the objective of the session should already be defined and a series of interview helper questions are prepared.
Interviews with existing users are usually more focused on the existing product and how it fits in with their needs.
We’re trying to understand if improvements or ideas we are going to develop will satisfy an unmet need. We tend to use the app as a basis for the discussion while already asking a set of questions to establish a baseline, such as: Are they a frequent user? Are they tracking on a specific category? Are they more interested in the app as a contraceptive or to get pregnant?
For potential users, it is to learn about their thoughts on the broader topic of health and what other similar, adjacent or complementary, products, and services they are using.
The key here is to understand why are the other products and services are useful or helpful to them. With this group, it is more likely that we uncover a fundamental difference in terms of feature sets or simply understand how they perceive the topic.
For Non-users, the interviews are completely different. We are going much, much broader.
We approach them more from a birds-eye view: How do they perceive their health? Would a digital health app be something they would ever consider at all? What are their barriers to consider using a service? Why would they not use such a service?
Non-user interviews tend to uncover new use cases for the app or service.
What are realistic goals to define before doing user interviews?
You need to have a hypothesis and a clear objective of what you want to achieve. You can’t just say ‘I want to do user research but I don’t know why.’
The more narrow the topic, the better it is. The less clear on the topic and objective, the higher the likelihood the conversation with the user could derail.
However, the good thing about user interviews is that we will always learn something new, but it might not be helpful with bringing a project forward.
Interview examples:
Will changing the layout of the calendar in Clue help with faster tracking?
- In this user interview, we target existing users to know if a different calendar layout can improve their overall tracking experience.
- We show multiple prototypes of different calendar layouts to users and collect their feedback. Prior to showing prototypes, we will ask how do they use and interact with the calendar.
What factors create a deeper trust when using Clue?
- Trust is a broad topic but it has a clear focus that is applicable to all three user groups; existing, potential, and non-users. This will also eliminate any bias.
- Results from each group are synthesized to understand the overlapping or distinctive behavior of each group.
- Additional research before interviews needs to be done to understand why certain apps and services seem to provide high levels of trust with their users.
If you would like to learn more about what kind of questions Clue is asking in user research, check out their blog post.
Could you talk a little bit more about quantitative and qualitative research? How do you decide when to go either qualitative or quantitative?
It has to be a pair, it can’t just be only quantitative or qualitative. Both are important because they have different biases.
Quantitative research is useful when you already have a specific feature in mind and just need to validate a specific area of that feature, such as content style or structure.
This would be something like running a survey — asking the user a series of questions so they have to choose something. They’re more suitable for deciding a feature priority or the way they are displayed.
Assume, Clue displays insights in the app and there are 15 different types of insights to show, then we could list these insights and ask users to order them in terms of preference. Qualitative research would not work as well in this case.
Qualitative research is mostly trying to get users to reveal motivations and preferences they don’t even know they have.
When doing qualitative research, we never ask the user a direct question like: ‘Which are your three favorite features that you use in the app?’. Instead, we will ask indirect questions that don’t target a feature at all: ‘Why do you do this? Is there a particular reason? What kind of value do you see in this thing?’
If we want to know if a user is willing to spend on the Clue app, we wouldn’t ask ‘Would you pay for the app?’. Users will probably say ‘yes’ and the team will still not know what kind of features or services that the user would pay for. Instead, we ask questions like ‘Which feature provided you insights to your body?’ or ‘What would be a feature you wish the app to have?’. Diving deeper into those topics can reveal what users would perceive as valuable and it will help the team to design features which align with this.
You need to be pretty good at improvising. When users are stuck, you should be able to ask similar questions in a different way.
With qualitative research, we get to have a glimpse of the lifestyle of the user and the way they think about the topic itself. For this, it is important for us that users feel comfortable talking and sharing.
So you want them to relax and forget they’re being watched?
Yes, because they have the bias of trying to be helpful. People generally have a strong desire to help. So they answer what they think we want to hear. The best way to get around this is by asking them to tell us about themselves and also subtly dive into the topic without being clear on the objective of the interview.
What should a good interviewer never do in user interviews?
As an interviewer, you shouldn’t be helping your interviewee. A lot of times during the interview users will be looking at the interviewer for signs. They want to get help or a signal if they are answering correctly.
Instead, make it clear to the interviewee that there is no right or wrong answer and ask support questions: ‘What are you thinking?’, ‘What are you looking at?’. If they’re interacting with the app and can’t complete a task, you could ask: ‘What are you looking for?’.
How many user interviews are enough to gather the information needed?
Usually, with qualitative interviews more does not equal better. With the right hypothesis, patterns start emerging after user number 5 or 6. We have tried running user interviews up to 20 users from a broad range of criteria, but after 10 users a clear pattern emerged and it was more reinforcing that users do perceive certain topics similarly.
How do you deal with inconclusive results?
A wide distribution of results can happen when questions, hypotheses, methodology, or supporting material are not well-defined. It is important to understand that the user is not at fault. You have to go back, revise, and iterate.
Any final advice?
User testing is one of the tools within a Product Manager’s toolbox. It is vital to understand that user interviews are an integral part of a successful product development.
There are many best practices. Not all of them might be applicable for you, but the journey to find a suitable approach to fitting your product will be beneficial in the long run.
Table of Contents
