Mitigating Risk and Maximizing Impact
“The only thing we have to fear is fear itself” — Franklin D Roosevelt
Notifications can deliver significant engagement uplift, but, if abused, may result in negative user sentiment. What we routinely see at Phiture, is that teams are far more likely to under-deliver on notifications impact due to fear of annoying users than they are to drive users away through over-communicating.
This is a follow-up to my recent post where I outlined a model for considering the impact of notifications; the RRF Framework.
In this post, I’ll expand on the topic of impact. The first part of the post deals with how to conquer fears and unlock greater potential from notifications:
- Testing ideas for notifications while limiting risk
- Dealing with fallout and mitigating negative effects
- Putting users in control
In the second part, I’ll also outline some useful concepts for building a notification system that auto-optimizes towards maximum impact and minimum user annoyance. At the heart of this smart system is a Notification Service that is informed by:
- User profiles, based on past behavior and predictive analytics
- External and internal events to trigger notifications
- A Prioritization Engine
I’ll lay out what exactly an adaptive, scalable and essentially stateless system for notifications could look like.
Notifications deliver impact
Relevant, timely notifications demonstrably improve retention; I’ve seen this for myself when my team built and deployed a real-time activity notifications system at SoundCloud. There are also plenty of reports out there that support this hypothesis (albeit produced by vendors of marketing automation tools, so some bias there).
Notifications won’t fix a poor product; in fact, they might amplify the flaws by bringing users back more often to a dismal experience. However, if the product is already engaging, notifications can usually help keep users around.
Push demonstrably works (source: Localytics)
The fear is often overblown
It’s certainly possible to increase user churn and opt-out rates for email and push notifications with poorly-designed notifications, but fear of these negative effects is routinely blown out of proportion, leading to a significant under-utilization of these powerful channels and leaving valuable impact undelivered.
Fear of notifications being perceived by users as spammy is endemic in teams building mobile products. What typically fails to elicit anywhere near the same level of panic is the huge chunk of users who abandon the product within the first 24 hours, with many more churning over the next few days. Given this scenario is the status quo for the majority of apps out there, you’d think teams would be reaching for the big guns in an effort to fight back this crushing tide of abandonment threatening to wash their products into obscurity. Notifications, when used wisely, can significantly improve the situation, but their potential is often muted due to fear of causing offense.
In actual fact, relevant notifications in a multi-user ecosystem can create an engagement ‘snowball effect’. This is best observed in social networks, where social interactions (likes, comments, etc.) generate notifications, which in turn generate new interactions, and so on.
The virtuous cycle of social notifications
How to test ‘risky’ ideas with minimal downside
If there are concerns about possible negative effects stemming from notifications, ensure such effects are properly accounted for when instrumenting the analytics around these notifications.
A data-driven approach, a robust testing framework, and judicious use of control groups are all that is needed to establish whether these fears are justified and provide the team the much-needed freedom to experiment and learn from mistakes.
The formula for ‘safe’ testing of new messaging strategies and notification types is simple enough:
- Test with a small (but significant) sample of users, ideally new users (many of whom will, as we have established, churn anyway, and hence have the most upside)
- Compare the results versus a control group
- Establish what impact was achieved
- Check if fears were justified (review negative signals such as opt-outs and churn vs the control group)
- Adjust course accordingly
When one considers that an average app loses 77% of its Daily Active Users in the first 3 days, testing bold new experiments on this group has very little downside: these users will churn if you don’t find a way to keep them around a little longer and give them another chance to have a positive interaction with your product.
There is often more upside than downside when it comes to reducing new user churn
If positive impact is achieved, consider broadening the test to all users, or examine the results on older cohorts (i.e. existing, engaged users), again vs a control group and with a minimum-viable sample size. If they behave more negatively, you can still roll it out to new users only and increase that critical early-stage engagement. In my experience, most decent, informative notifications drive impact across new, engaged, and lapsed users, albeit to differing extents.
Mitigating Fallout from Negative Effects
Testing and measuring for negative signals such as opt-outs uninstalls and drops in engagement will let you know what proportion of users are turning their backs on notifications, or the product itself. The critical optimization problem to solve is how to mitigate such negative effects among users who dislike your notifications without slashing the overall impact by dialing back notifications in general, even to those who would happily engage with more of them. Most companies do not solve this problem; those that do are able to unlock additional growth.
First and foremost, notifications should be designed with the user in mind and should be timely, relevant (ideally personalized), etc. rather than generic broadcast messages to the entire user base. These considerations were covered in my original RRF post.
For now, let’s assume that the notifications are at least ‘decent’, as evidenced by the significant positive impact on engagement KPIs among users receiving notifications.
In the remainder of this article, I’ll outline three approaches that will mitigate negative effects associated with notifications in ways that don’t sacrifice impact with the general population of users:
- granular notifications preferences
- explicit and implicit subscription
- dynamic user-level optimization
Putting users in control
Writing for Wired, personal technology expert David Pearce recently made a hyperbolic case for turning off all push notifications. Pearce argues that it’s time for users to take back control of their attention and fight back against what he refers to as ‘commercials’ being inserted into users’ lives. I would argue that only poorly-designed notifications with very low relevance seem like commercial interruptions, but I do agree with Pearce’s core argument that users cherish a feeling of control over their digital lives.
This is what 8 million Instagram followers does to your notifications
Smart systems mitigate risk, rather than avoid it and lose the upside
A fearful approach to the risk of annoying users would be to adopt a very cautious approach, for example, a strict rate limit which caps the maximum number of notifications that a user will ever receive within a certain period. Such methods are crude and leave a lot of potential impact untapped.
In the next section, I’ll outline some nuanced approaches that can be taken to mitigate the risks of opt-outs and negative sentiment, then go on to describe a system that can both mitigate risks and capitalize on the — frequently untapped — upside from highly-engaged users.
In-House Notifications Preferences
Providing users a more fine-grained way of controlling which notifications they receive serves two purposes. Firstly, it allows users to opt-out of specific categories of notifications without opting out at the system level (thus keeping the channel available for other types of messaging). Secondly, opt-outs from inside the app are more easily tracked by analytics and hence provide a valuable feedback loop; if opt-outs for a specific notification category suddenly spike, it’s a valuable and timely indication that something is wrong and affords the developer a chance to fix it. When users opt-out at the system level, it can be tougher to track this in a timely fashion.
The Pinterest app (shown below) provides the user extensive control over the kind of notifications they receive, with preferences for both email and push, from the settings menu inside the app.
Pinterest gives users a comprehensive list of notification categories to control
With the advent of Android 8.0 (Oreo), similar functionality is provided at OS-level, through a new feature called Notification Channels, which should obviate the need for a more custom UI. However, Android’s notoriously slow upgrade cycles mean that Channels won’t be ubiquitous for a long time and iOS notifications preferences at the system level are still binary ON/OFF, so it’s probably still prudent to build and maintain an in-house notification preferences system for the time being.
Android 8.0 introduces ‘Notification Channels’, allowing users to customize preferences for specific categories (‘channels’) of notifications
Explicit Subscription
Another way to put users in control of the notifications they receive is to give them features that allow them to request to be notified when things happen regarding specific comment threads, newsfeed items, content, or users.
This is next-level granularity since it allows users to be ultra-specific about specific content that they would like to be notified about, as opposed to broad categories, or a single global opt-in for all notifications. Consider the die-hard deadmau5 fan, who wants to be notified whenever he drops a new track or someone who wants to ensure they don’t miss a single update on a conversational thread… explicit subscription mechanisms allow them to do just that.
Implicit Subscription
The problem with an explicit subscription is that it often remains something of a power-user feature; it can be tough to drive awareness and mass-adoption of such controls, even though users might have appreciated receiving updates about certain things they are interested in at a higher frequency had they discovered the subscription feature.
A smarter notification system might begin implicitly subscribing users to certain content based on their behavior history, obviating the need for them to explicitly hit the subscribe button but making it clear that they can unsubscribe.
Such auto-subscriptions could be processed periodically after e.g. a user has listened/watched content from a particular artist/read posts from a particular contributor enough times for the system to have a high degree of confidence that this is one of the users’ favorite content creators or content types.
If YouTube would auto-subscribe users to channels after they had watched 10 of its videos, I predict few users would complain (since they have implicitly expressed a clear preference for the content) and subscription rates would be significantly higher. Users could still unsubscribe using the button. The upside for YouTube and the creators on the platform would be huge in terms of additional engagement generated from fans who would never have been bothered to press the subscribe button but love the content nonetheless.
Blueprint for a smarter notification system
Notification preferences and subscriptions are great for giving the user more control, but wouldn’t it be great if users never felt the need to adjust their notifications settings? Imagine building a system that is smart enough to adjust each user’s mix of notifications according to their changing tastes and their engagement with notifications.
A smart notification system adjusts to users’ engagement with previous messages and uses behavior history to estimate the relevance of notifications before deciding whether to send them. Events shown are just examples; these would be specific to the app/platform.
The system described takes inputs in the form of events occurring on a platform. The example used is a content platform: this could be a music streaming platform like SoundCloud, a video streaming service such as Netflix or HBO, a blogging platform like Tumblr or Medium. Any platform where there is a lot of content being produced/released and consumed would be a good fit for such a notification system.
The system has the following basic properties:
- Each notification type has a rate limit (aka Frequency Cap): a frequency which under normal circumstances will not be exceeded e.g. one notification per 24hrs.
- If more than one channel is supported (e.g. Email as well as Push Notifications), rate limits are also channel-specific; Push rate limits can be different to Email rate limits for the same notification type.
- Rate limits are calculated and adjusted on a per-user level by maintaining a user Engagement Score (more on this later)
- Rate limits can be exceeded (more notifications of this type are sent within the rate limit window) in exceptional circumstances. Exceptions are decided by the Prioritization Engine, which estimates the relevance of each notification to a specific user; if the relevance estimate is high enough, it‘s likely worth exceeding the user’s daily maximum.
Stateless system = cheaper and faster to deploy
One advantage of the system outlined above is that it is essentially stateless; it processes each event in real-time, delivers notifications more or less instantly, and does not incur the significant computational and storage overhead required for buffering/caching events and notifications. This means that such a system can be built and deployed faster… in the retention team at SoundCloud, we deployed an MVP version of such a system within three months of kicking off the project. While we made numerous enhancements over time, the system delivered very significant impact from launch day onwards.
Aggregation of similar notifications requires a more expensive approach
While the system described is one that can be built and deployed relatively quickly and cheaply, a more sophisticated system would cache events as they come in and maintain queues of notifications at a per-user level. Such a system would be able to choose between competing notifications (selection) and aggregate notifications of a similar type (e.g. “Your post received 5 likes”).
Aggregations are undeniably powerful, but this functionality has been omitted from the above system since they introduce considerable complexity, both in terms of system design and implementation. When buffering and aggregating notifications, the designer of the system needs to consider such issues as timeliness, batch sizes for aggregations, how long to wait before sending the next notification to a user if no new events are received (but qualified events are buffered), etc. To avoid this additional complexity, the remainder of this article will focus on the simpler, real-time, stateless system outlined above.
User Engagement Scoring: enabling self-balancing, personalized notification limits
Imagine an app that could dial up or down the frequency of notifications based on your personal engagement with them. Ignore your notifications, and it sends fewer, less often. Click on a bunch of them and it starts sending you more of the things you’re interested in.
By employing a Machine Learning approach, or simpler algorithmic method of adding and subtracting points based on interactions (or lack thereof) with past notifications, the system dynamically adjusts the volume of notifications each user receives (i.e. adjusts their individual rate limit).
Why bother with Engagement Scores if the app already includes granular notification preferences?
There are a couple of reasons why maintaining an engagement score might be a good idea, even if a comprehensive set of preferences are implemented in the app settings menu:
- Engagement Score is automatic and based on inference rather than explicit user action: users do not need to explicitly find this feature and make manual changes. Thus, all users benefit from it, not just the users who find and use the manual settings.
- Even fine-grained preferences typically only allow the toggling of specific notification categories on/off. An Engagement Score allows every on the possibility to be more on and reduces the likelihood that anything will be turned off since it will be dialed down to a lower frequency by the system.
- While, in an ideal world, the engagement score system works so perfectly that it obviates the need for explicit preferences entirely, there will still be some users who enjoy the feeling of control that a full set of preferences provides, much as some people would prefer a steering wheel and pedals in their self-driving car, just in case they feel the need to take over.
The Prioritization Engine: exceptions to the rate limit for top-priority messages
Think of a Prioritization Engine as a sorting machine that tags message-user pairs with a level of importance, and thus influences the decision on whether to notify them with this specific message, or leave them undisturbed.
The Prioritization Engine’s job is to guess — as best it can — the relevance of a specific notification to a specific user. A really smart system would employ machine learning to become progressively smarter, based on a feedback loop in the form of open rates and conversion data from previous notifications. The aim here is to discard notifications that are unlikely to resonate with the user before they are sent while flagging those that have a great chance of being engaging, so they can be prioritized and possibly even override the standard rate limit established for that user, to send a notification that would otherwise have been prevented by the rate limit.
While it sounds fancy and complex, an MVP prioritization engine could employ simple heuristics (basic rules of thumb/logic rules) to try to identify the ‘best’ or ‘worst’ notifications for a specific user. As long as it succeeds in improving, on average, the ratio of open to unopened notifications (i.e. increasing the average relevance of notifications) that the system delivers, it’s adding value to the system.
Improving on simple logic rules, the Prioritization Engine could leverage analytics data about the user’s historical engagement. For example, pre-calculated ‘favorites’ lists (favorite content, favorite creators or publishers of content) based on the usage history of each user and prioritizing notifications about content or creators the user has expressed a clear predilection for.
The logic goes that notifications about content/creators/product categories that the user has interacted with a lot in the past will be welcome at a higher frequency than of those that had received little prior attention from the user.
The danger of this kind of prioritization logic is that it can create an echo-chamber effect, where new content is not surfaced due to a bias toward the already-discovered. This effect could be reduced by only applying prioritization to notifications once the rate limit has been reached, to decide if a new one should be permitted — if predicted to be sufficiently relevant — to bypass the rate limit.
Conclusion
- Notifications usually fall short of their true potential due to fear of overusing them
- Careful experimentation with control groups is necessary to establish ‘how much is too much’. Test first with new users, where feasible
- Giving users explicit control over the kind of notifications is good practice and also provides valuable information
- Inferring user tastes from their behavior may unlock further potential for relevant notifications
- Dynamically adapting the volume of notifications sent on a per-user level, based on prior engagement and/or predicted relevance, can unlock further impact
I would love to hear from growth practitioners what is working well for them in terms of optimizing notifications impact. If you are willing to share your experience, I’d appreciate hearing from you. Likewise, if you have comments or feedback on the concepts outlined above, please get in touch or leave a comment below.
Finally, if you’d like help with building such a system for your business, feel free to reach out to my consultancy firm, Phiture.
Table of Contents
