Why Prioritizing Your Agile Backlog for ROI Doesn’t Work

Pundits tell us to prioritize our backlogs to generate the best possible ROI. But no agile teams that I’ve ever worked for or with do this. So the pundits must be wrong, because Agile continues to provide stellar results for a lot of product teams.

Backlog Prioritization 101

In its simplest terms, your backlog is the discrete set of deliverables your team must create. Regardless of your particular flavor of Agile, all successful Agile projects (and, truthfully, all successful projects, Agile or otherwise) rely on the effective prioritization of work. To prioritize effectively, product managers must identify a set of attributes for backlog items that will be used for prioritization, assign them a value, and then sort the list based on the values of these attributes.

The pundits say that you should value your backlog based on ROI, NPV, net cash, time for payback, or any number of other financial metrics that boil down to comparing revenue vs. costs for individual backlog items. In all of the years I’ve been doing Agile, I’ve never seen a product backlog where each backlog item slated for the release has an associated ROI analysis. Heck—I’ve never seen a backlog where more than a very few items have an ROI (this does not hold true for portfolio backlogs, which are often prioritized for ROI based on sensible market analysis).

Product backlog items that do have revenue associated with them tend to be directly associated with contractual obligations, often referred to as Non-Recurring Engineering (NRE). This kind of ROI is simple: customer so-and-so is willing to pay $x for this feature, it will cost $y to implement it, and since $x is >= $y we’ll do it. Except that in quite a number of cases product teams will do it even is $x <= $y, because customer so-and-so is either a really big customer, or they rationalize that if customer so-and-so wants it other customers will want it, or that they need the revenue, or, because only these items have ROI calculations, they bubble up to the top because they have more data, regardless of whether they are the best choice.

When the performance gap between espoused theory and actual practice is large, you need to look harder at both to understand what’s causing the gap. Sometimes, practioners are simply “street-smarter” than their more academically motivated counterparts, and the espoused theory has to change to better match the realities of practice. At other times, the practice is just perpetuating old-thinking, and the theorists have a better approach. In this case, I think that street smart Agile practitioners are smarter than those Agile pundits who spend most of their time teaching theories instead of learning from practice.

The Problems With Prioritizing Your Backlog Based on ROI

There are several reasons why prioritizing your product backlog based on ROI is problematic. Here are just a few. Feel free to add your own.

• To establish a realistic ROI, you need to engage in appropriate market research. Unfortunately, most market research that can establish realistic ROI, such as a conjoint analysis or a willingness-to-pay study, is usually based on relatively large chunks of functionality that may or may not correlate to what is on the backlog. The unfortunate side-effect is the release planning process, in which the development team negotiates the actual features with product management, can result in changes to the tested features that are so large the original research is invalidated.
• In attempting to solve this problem, many Agile teams are adopting the Incremental Funding Methodology (IFM) popularized in the book Software By Numbers by Denne and Cleland-Haung. Simplifying quite a bit, IFM is based on using market research to establish Minimum Marketable Features (MMF) that can be used to maximize NPV. While I love this approach in theory, because it celebrates direct customer input, I haven’t found any examples of this being used in practice. I suspect that one reason is the method, while relatively easy to explain, has a significant amount of overhead, and defensible NPV needs to be based on defensible revenue data.
• The IFM also suffers from the same problem as much traditional market research in Agile projects: the backlog items tested are created before release planning, but the backlog items implemented are changed during release planning. This isn’t to say the concept of market research in an agile project is fatally flawed. It isn’t. We need more market research in the Agile community, not less. We just need market research tuned to Agile projects..
• A related challenge is that Agilists prefer to manage relatively small user stories, but customers tend to value groups of related and often interdependent stories that together solve complex problems (something Jeff Patton refers to as “task” focus). And while any good Agile Product Manager hates interdependent backlog items, the reality is that sometimes backlog items really are interdependent. To illustrate, one of my clients was building regulatory compliance software in the trucking industry. Many of their customers had large operations in the US and smaller operations in Canada and Mexico. The NPV for the MMF “provide compliance software in all geographic markets” would be positive, but the NPV for the MMF for each individual backlog item (one for each market) would be negative.
• Creating, executing, and analyzing a statistically significant market research project can take longer than a release cycle for a good agile team. In one client project, I participated in a comprehensive conjoint analysis for a rather large system. In this case, it was the right choice for the client, but the overall project took about 7 months—far longer than the release cycle of most Agile teams. And the expense ($100K’s) means that it is highly unlikely any company would fund more than one of these projects per year.
• Statistically significant market research may be antagonistic to Agile. Consider the co-creation of value, which involves active participation of companies and their customers, is evolving to the point where many market segments don’t expect to be asked to respond to traditional surveys. Instead, they expect to be allowed to collaborate with the company—and their customers—to jointly develop specifications and solutions to complex problems.
• Agilists celebrate our ability to respond to change rather than following a rigorous plan. Which sounds great until you’re asking a product manager who just invested a lot of time and money in her market research to reshuffle the backlog based on new information. They’ll be torn. The market research says they should stick to the plan. The new information says it should change. The greater the investment in the market research, the more challenging it is to make the change.

I suspect that even with all of these problems, pundits continue to recommend this approach because it sounds easy, it creates long-term, and therefore, expensive consulting projects, and it makes Agile easier to sell to MBA-trained senior executives (“See? We’re prioritizing by business value, in terms that you’ve been trained to understand—NPV. Just don’t ask me to show you how I got that number, OK?). And, since I don’t see much emphasis on rigorously testing if the projected NPV matches the actual NPV, and what to do if they don’t, I’m admittedly skeptical of just how well ROI based prioritization methods perform over time (which is, if you’ll notice, the core aspect of NPV; money sooner is better).

Consistently prioritizing your backlog for ROI is a fool’s errand.

Backlog Prioritization Attributes

To effectively prioritize a backlog a product manager must identify a set of attributes that will be associated with backlog items, assign the attribute a value, determine the degree of influence this attribute/value should have (the weighting) and then sort the list based on these results. The following represent some attributes that have helped prioritize backlogs. Of course, not all attributes are used in any given project, and that the acquisition of data needed to support a given set of attributes is itself a cost that must be included in the overall analysis.

• Customer preferences as established through various market research methods
• Cost-based financial attributes, which come “for free” as part of the various estimating processes used by Agile teams
• Perceptions of internal stakeholders (sales, services, development)
• Corporate priorities, such as a preference to develop a new market or improve operational efficiency
• Degree to which this backlog item is dependent upon external systems outside of the control of the development team (especially useful when more Agile teams are working with less Agile (“Waterfall”) teams
• Various kinds of risk attributes, such as market or technical risks
• Attributes that drive the profit

Most product teams create groups of related attributes, a sensible approach that enables them to focus attention on a “slice” of the problem without becoming overwhelmed.

I strongly recommend that before reading further you review the attributes you’ve developed to prioritize your backlog. How are they working for you

Choosing Backlog Prioritization Attributes and Weightings

While the exact set of prioritization attributes and their associated influence (weighting) on the sort order of the backlog varies on a per-product, per-release basis, I’ve found that product managers work most efficiently when the backlog attributes are relatively stable while the weightings of these attributes are adjusted to reflect current thinking. As such, a product manager should select prioritization attributes that meet the following criteria:

1. They are valid across a few releases
2. Their value can be established in a timely and cost-effective manner
3. They support the politics of effective product management
4. They align the backlog to corporate and product goals

I’ve found that the following sets of attributes meet this criteria on every project:

1. Stakeholder Preferences
2. Strategic Alignment
3. Driving Profit

I also prefer market research that establishes direct customer preferences for backlog items, but, quite frankly, it is the rare product team who does this. That’s why the third set of prioritization attributes is so important, as these can help a product manager compensate for the lack of direct customer feedback.

Stakeholder Preferences

Stakeholders come in two primary forms—internal and external stakeholders. Internal stakeholders refer to sales, marketing, professional services, and customer service. Their perspective on the backlog is critical for success, because these groups must work well enough to create a successful project. A critical addition to this list of internal stakeholders is your “System”. Here’s why. Every system architecture, no matter how well-designed, must be maintained. This can be a challenge in agile teams who focus their attention on backlog items that are associated with top-line revenue growth and/or customer demands. To combat this tendency, I advocate anthropomorphizing the system . This enables the system to literally “have a voice” in its own “care and feeding” and helps encourage agile teams to make architectural investments that are every bit as important to the long-term needs of the business as various features are to other customers.

External stakeholders include customers, channel and marketing partners, integration and/or development partners, distributors, and quite possibly the general community that is associated with your product. In the ideal world you’d include their preferences directly through market research (Innovation Games^®, Min-Max, conjoint analysis, Kano analysis, and so forth). In practice, most product managers include their preferences indirectly by interpreting their impressions of what customers want.

Strategic Alignment

The backlog must also support the larger corporate, divisional, and/or portfolio goals established by the executive team. This means product managers work with the executive/business team to make certain the business priorities are clear and weighted according to the goals of the business.

To illustrate why this is critical, several years ago I was asked to help a product team adopting Agile find a way to manage their boss, who was inadvertently micro-managing the prioritization of the backlog. Specifically, after the product manager had established the backlog prioritization, he would inevitably change it, dis-empowering the product manager and dramatically slowing down the project. He didn’t mean to dis-empower his product manager. Instead, he was frustrated by his inability to see how the backlog was prioritized relative to his larger divisional goals. Once we added attributes that captured the divisional strategy, he was able to see which backlog items explicitly supported the same.

Driving Profit

I believe your business model is the system of inter-related choices that you make to drive profit. Two important components of this system are your value exchange model and your profit engine. Your Value Exchange Model is how you trade money for value in software. One example is a Time Based Access model, in which a customer pays you money to use your software for a period of time. Perpetual and annual licenses are the most common, with Adobe’s Acrobat being an example of the former and many enterprise software offerings being an example of the latter. Another example is a Transactional model, such as the credit card industry’s transaction processing models, Google’s model for revenue based on advertising or Enthiosys’s model for Innovation Games^® online. (For a detailed exploration of Value Exchange Models, see my book Beyond Software Architecture).

Your Profit Engine is how you structure your relationship with your market to drive increasing exchanges of value. In Adobe’s case, they make Acrobat reader free and ubiquitous, driving the growth of the writer. In the case of the credit card industry, they make their networks widely acceptable, drive common standards to lower certain kinds of costs (the shape of the card, the contents of the magnetic stripe, and card readers), and offer additional services to drive transactions (anti-fraud and sophisticated account management services).

Unlike ROI, every Agile Product Manager should be able to identify which backlog items contribute to driving profitable growth. For example, if your value exchange model is based on transactions, a backlog item submitted by your technical team to reduce transaction processing time by 10 ms per transaction drives profit by lowering operational costs. If your profit engine is based on selling add-ons or modules to a platform, you’ll want to know which backlog items drive the growth of existing modules or create new ones.

A key distinction here is these attributes are NOT designed to capture improvements to existing solutions. Instead, these attributes are designed solely to align backlog items with those aspects of your business that drive profitable growth: lowering operational costs, driving top-line sales, motivating existing customer to purchase more of your total offering, and so forth.

My experience is that successful Agile Product Managers have an almost natural, intuitive feel for prioritizing backlog items to drive profitable growth. Fortunately, this natural ability is greatly enhanced when they make the link between the backlog and their ability to drive profit explicitly. As an aside, it is this set of attributes that most clearly distinguish the Scrum-centric role of a Product Owner with the full responsibilities of an Agile Product Manager.

Setting Prioritization Data

Prioritizing against the three sets of attributes above means that Agile Product Managers must indentify stakeholder preferences, understand which backlog items align to corporate strategy, and demonstrate which backlog items drive profit. Each of these involves gathering a unique set of data.

Gathering Stakeholder Preferences

The ideal way to gather internal and external stakeholder preferences is to collaborate with them to get their perspective on backlog items. The kind of collaboration we advocate requires product managers to move beyond the many traditional forms of market research (which, as outlined above, may take too long or cost to much) and embrace new thinking and new tools.

One approach is to leverage preference weightings and collaborative gaming to enable continuous collaboration on the prioritization of backlog items. Focusing on collaborative gaming, Agile Product Managers leverage in-person or online tools to collaborate with groups of stakeholders to gather requirements.

For example, a product manager might use the in-person version of the Innovation Game^® 20/20 Vision to rank-order the benefits that a small group of customers (typically 8-30 lead customers) want in the next release and the online version of Buy a Feature to gather direct feedback from an arbitrarily large number of customers (or other stakeholders, such as channel partners) on the backlog items that represent the features that create these benefits. Using Buy a Feature online, they could gather and compare external stakeholder results with the results of various groups of internal stakeholders. These collaborative approaches compliment and extend the personas that astute product managers create to drive success of their products. As you can expect, the highest priority backlog items will be preferred by both internal and external stakeholders.

This process can inadvertently prevent Agile teams from prioritizing architectural or system level enhancements into a release. I know, because I’ve made that mistake. To compensate for this, recall that we anthropomorphize the system architecture to capture necessary architectural changes from the roadmap, with the technical architect representing the “voice of the system”. Thus, even though no other internal or external customer cares about refactoring or improving your architecture, it can still be prioritized high on your list because your system cares. As a result, every backlog item should correlate to at least one stakeholder preference.

Aligning to Corporate Strategy

Aligning your backlog to your corporate strategy is simple – provided you know your corporate strategy! Assuming you do, simply associate corporate strategy attributes to backlog items, identify which backlog items directly support that corporate strategy, and demonstrate that at least some of these backlog items are contained within your release. For example, a 2009 strategy of focusing on developing solutions that are mobile, global, and social. An Agile Product Manager here would seek to show which of their backlog items directly align with at least one of these strategic initiatives. Not every backlog item will align to corporate strategy, which is OK (sometimes a bug fix is just a bug fix).

Product managers who find it difficult or impossible to align their backlogs to their corporate strategy need to tackle some potentially serious problems. When the root cause is an unclear corporate strategy, it is the product managers responsibility to suggest their best possible interpretation and encourage senior leaders to extend and improve the same. When the root cause is a backlog that is challenging to align with the strategy, I suggest exploring a change in the backlog or a change in the strategy. The former is appropriate when the backlog was developed without awareness of the strategy or when the backlog was developed under the context of a previous strategy. In this case, the backlog should be refreshed under the context of the current strategy, which is likely to motivate a new round of market research.

There are times, however, when a product manager is faced with solid market research and an associated backlog that justifies a change in strategy. In this case, the product manager should negotiate with senior leaders to explain their evidence, demonstrate how the backlog aligns with a modified strategy, and make the case for the updated strategy. Sometimes the right approach is to indeed change the strategy.

Driving Profit

Stakeholders, and especially existing customers, may want (value) a large number of backlog items that they expect “for free” because most software companies have conditioned their markets to expect continuous improvements to products and services as part of existing business models. While implementing these may improve customer satisfaction, they may not drive profit. Similarly, the product manager who can demonstrate that a backlog item aligns to the corporate strategy of providing a mobile solution may be hard-pressed to identify how this solution will drive profit. Again, this may be OK, as there may be larger corporate goals served by developing a mobile solution in a given product even though that particular product may not be able to directly monetize that profit.

Eventually, though, a product needs to contribute an appropriate profit to the firm, and it is the product managers responsibility to demonstrate which backlog items are expected to drive profit. For example, suppose that you’re managing a hosted service and have some backlog items motivated from your operations team that identify ways in which you could reduce operational costs. These items serve to drive profit. And yes, I know that because computing the NPV/ROI for these items is relatively straightforward you’ll be tempted to do the same. Go ahead, if you must, but remember: prioritizing your backlog solely on ROI is not your goal because you’ll not be able to compute the ROI on every backlog item.

A more realistic (and therefore, complex) example occurs when your market data (qualitative market research, win/loss analysis, competitive offerings, etc.) results in backlog items that you believe will enable you to sell more of your software, perhaps to existing customers or perhaps to a new market segment, but lacks the necessary rigor or completeness to support a defensible NPV/ROI analysis. In this case, you’ll want to identify they key ways in which this backlog item will drive increasing profit. And, if you’re working in a truly agile company, you can also provide some broad insight into how much revenue you’ll make.

For example, suppose you’re developing a content management system for patent data. This is a market that values data. Lots of data. The more data, the better the analysis. Suppose your current offering manages data from the European and US patent offices. You’ve gotten some requests to manage data from the Japanese patent office, and you know some prospects have listed a lack of Japanese patent data as a reason they didn’t buy your system in their win/loss analysis. In this case, you can legitimately claim these backlog items are going to drive revenue through new sales, even if you lack the data to complete a defensible ROI calculation (e.g., how many customers will pay the necessary amount to drive growth or how many customers will pay a data management upgrade fee to acquire this new data). If this backlog item further aligns with a corporate strategy of managing large amounts of patent data, you may have sufficient data to prioritize this backlog item into the release, provided you accept the risk that you didn’t justify your choice with a defensible ROI analysis.

In practice, I’ve found backlog items that drive profit tend to be rather large amounts of work, often called epics, that during release planning are elaborated into many smaller backlog items that can completed in an iteration. That’s OK, because the benefit of enabling Agile teams to clearly define how they are driving profit through their backlog outweighs any issues.

A Well-Prioritized Backlog Drives Profit

Now that the core attributes have been identified, grouped in sensible ways, and set their values, let’s examine the results of the work to determine if the backlog items that have made it into the release are likely to drive profitable growth. Instead of prescribing more processes to follow, I’ll instead provide three simple tests that you can apply to your backlog to help you determine if you’ve done a good job (feel free to add your own).

Test 1: Stakeholder Balance

A stakeholder balanced release backlog has at least one backlog item for every stakeholder in the release. Yes, everyone. That’s because Agile Product Management is a political game, and you’re not going to succeed unless you manage the politics of the backlog. Which means you’re going to eventually have to prioritize your support teams request to improve the log files so that they can more effectively diagnose customer problems.

An unbalanced release is a release that starves one or more stakeholders. You can get away with starving a stakeholder every now and then, but consistently starving an external stakeholder will eventually lose you customers, partners, or your channel. Consistently starving your internal stakeholders means that you’ll lose the goodwill that results in sustained success. Your sales team may not push your product quite as hard, support may not support it quite as well, and so forth.

Test 2: Strategic Alignment

A strategically aligned release backlog has at least one backlog item that can be directly correlated to the key elements of corporate strategy. Hopefully, I don’t need to elaborate the long term job prospects of a product manager who consistently ignores or is unaware of how their product’s evolution correlates to product strategy.

Test 3: Profit

A release backlog that is driving profit has at least one backlog item that can be shown to be congruent with your Value Exchange Model and drive profit through your Profit Engine.

In summary, the key to prioritizing for profit is to:

1. Balance stakeholder demands
2. Align to corporate strategy
3. Drive your profit engine

Learn more about the role of Product Management in agile environments at Pragmatic Institute’s Build.