In their seminal article on product management, Good Product Manager, Bad Product Manager, Ben Horowitz and David Weiden argue that product managers should be top-level leaders who set the vision and are ultimately accountable for their product’s success or failure. In essence, they should be CEOs of their products. Almost 20 years after it was written, it’s remarkable that their argument remains relevant in an industry defined by constant change. In reviewing the role of product manager as mini CEO, the role of data in decision and strategy support naturally surfaces.
New digital technologies throw off huge volumes of data that increasingly underpin key planning and decision-making functions. In fact, 89 percent of U.S. respondents to PwC’s 2015 CEO survey said that new digital technologies created “very high” value through data and data analytics.
But what about product managers? While companies have created standardized dashboards and tools for financial, operational and marketing functions, they have lagged in instrumentation for the products themselves. This leaves product managers at a relative disadvantage to the C-suite
and fails to adequately capture a tremendously valuable source of data.
Many product teams rely on web analytics for information about their users. However, these tools aren’t optimized for registered, logged-in users interacting with a full-fledged software product, especially one built around increasingly popular single-page UX designs. They need to understand how users interact with specific features and to map complex user paths through the application. Attempting to contort a tool that’s designed for mapping anonymous visits and optimizing sales funnels to meet a product manager’s needs is a difficult task and often results in suboptimal measurements.
To function as the CEO of their product and team, product managers must have a data strategy that meets their needs. They need to take charge of this strategy and ensure that their applications capture the data they need.
Events: The Building Blocks of Product Data
Before diving into specific measures, it’s worth spending a few moments talking through the basic foundation of product data events. An event is an action that a user does in an application. An individual event has several aspects: the action itself, the user who initiates the action, and the date and time that it occurs. For example, a URL page load (something that a web analytics package would track) is an event, but so are more granular actions like following a link or clicking on a page element. These basic events and their associated attributes are rolled up and analyzed to arrive at the key measures discussed in this article. As a part of any data strategy implementation, product teams must ensure that they can accurately capture these events in totality.
Key Product Metrics
So, what should product managers measure and how is it different from other key performance indicators deployed in the enterprise? These five key factors can provide unique insight into a product’s health.
Factor 1: Breadth of Usage
The simple way to think of breadth of usage is to answer the question, “How many users do I have?” But in reality, it’s more involved than that. Product teams should first benchmark their total users based on the size of their target market and then filter based on active users, not registered users. What constitutes an active user can vary depending on the application; and a product team must define what an active user means to them. At a minimum, there should be some sort of activity at least every 30 days.
It’s also important to consider the distinction between users and customers. For many SaaS companies, a single customer account will have multiple associated users. Product teams will want to measure breadth at the user and customer levels, and within customers as well.
Factor 2: Frequency of Use
Frequency of use goes hand-in-hand with breadth and focuses on how often and how long users engage with a product. Product teams should look at the average number of sessions within a given time frame and the average duration of each session. Combined, breadth and frequency provide a clear view of a product’s total use. A wide range of active users indicates an engaging experience that delivers differentiated value to a target audience.
These measures should be reviewed for the total user base, but it can be helpful to apply the same analysis to user segments. For example, the usage levels for premium versus basic subscribers—or across different user roles—are important cases that should be considered along with an overall measurement.
Factor 3: Depth of Use
Conceptually, this is a relatively simple measurement:
What percentage of application features get used within a specified time frame? However, in practice it can be a bit more complex and difficult to capture. Obviously, every user is not going to use every feature in an application. For effective measurement, product teams should look to first define their key features, the ones that make up the product’s core functions or are heavily used by happy customers.
Next, the feature measurements should be defined. Product teams need to identify which sequence or group of events constitute their key features and then aggregate the measurements accordingly. This factor is a strong measure of the value users receive from a product. If the majority of users only use a small subset of key features, that’s an obvious concern. However, even if the key features are widely used, groups of features that get little or no use could represent poorly used development resources and an opportunity to reprioritize projects.
Possibly an even bigger concern is when different users exclusively use different feature subsets. This can indicate a product without a strong core identity or value proposition. It’s often helpful to apply the same types of segment analysis to this factor, along with breadth and frequency measurements.
Factor 4: Efficiency of User Actions
How difficult is it for users to complete common tasks in an application? For example, think of adding a new vendor in a procurement application or posting a job requisition in a recruiting application. A common task represents a core function within the application and typically comprises several application features used in a specific sequence. To measure efficiency, product teams should look at the total number of users who begin a task and see what percentage successfully complete it.
This measure can augment user testing and help show the usability of key product functions. Understanding that some applications are necessarily more complex than others, product teams should still focus on driving high completion rates for the application’s core tasks. Looking at each step can help identify which specific feature causes users to bounce out of a process and identify areas for improvement.
Factor 5: Satisfaction and Qualitative Feedback
Satisfaction and qualitative feedback may seem odd to include, because by definition, qualitative feedback isn’t analytical. However, this is a significant source of insight for product teams that should be included in any evaluation of a product’s health.
Aggregating user events can give a clear picture of what users are doing in an application, but not necessarily why. Soliciting direct, open-ended feedback from users through polling or user testing can provide critical context to observed behaviors.
Qualitative feedback, whether through a standardized measure like net promoter scores or another mechanism, can provide an overall user-satisfaction rating that is a key indicator of growth potential. Product teams should always include this information as a part of any product data strategy.
When making strategic decisions, these five factors help provide a clear, detailed view of application health. They help prioritize roadmap decisions, identify break points in the user experience, and understand the behaviors of the highest-value and most satisfied users (who are not always one and the same). Most importantly, at this level, product data provides a consistent baseline of valuable information that product managers and their extended teams can use to support product decisions. By creating and incorporating a data strategy, product managers can set their vision to successfully function as the CEO not only of their products, but also of their teams.
