Many product managers like to work in startup companies. I’ve been one of them. Why? Without doubt, a product manager in a startup has an opportunity to make broad-ranging and meaningful contributions to product success, fast. But, life in that lane is not easy.
Early product definition tends to be quite elastic, leaving room for creativity. Corporate structure in a startup tends to be flat, allowing exposure to serious and far-reaching business and technology problems that impact decision making. That’s all good. However, one of the major difficulties is confusion. It’s all in flux. Yes, broad-stroke formulations of product vision and business concept do exist, the ones used to raise funds for the startup. But almost always, even these fundamental statements change with time. Flexibility is good but it’s hard to succeed without a firm grasp of reality and clear articulation of objectives, decisions, success factors and progress metrics.
How do you deal with this “mess” of moving targets and reference points? The tool that has not been broadly used but can be of great help is the product management audit. What is a product management audit? Let’s take a closer look, but first, I’ll highlight key issues related to product management in a startup. Then, we’ll define the specific problem we propose to address here and explore an example of an audit meant as a remedy for this problem.
The problem
The three issues that hold keys to product success and frequently, create problems in a startup are strategy, market and customer needs, and product requirements.
Strategy is extremely important for any endeavor, and especially for a startup, both the corporate strategy and the product strategy. The issue here is two-fold: Is our corporate strategy really sound and clear—as articulated—and is its impact on product strategy well understood? Do all key people in the company understand and agree on the current articulation of product strategy?
Target Market and Customer Needs define, almost completely, product requirements. Thus any lack of clarity here will soon lead to difficulties with the product. Have we fully decided who our target customers are and truly verified the scope of overlap between what they need and what we plan to deliver? It’s amazing how often the answer is: “Sort of, but not really.”
Product Requirements are key for a startup, driving both development work and early product promotion. No it’s not a matter of a polished text document. It’s a matter of having thought through and decided on the kind of product we aim to bring to market, and when. And, it’s a matter of mutual understanding among all the managers in the company what these requirements are. Are we building what we think we’re building?
So are these three issues the problem we address here? Not quite. Entrepreneurs usually know these issues are important and create problems if not addressed. However, in a startup, people are very busy doing a lot of different things. Moreover, entrepreneurs want to be optimistic, driving to success by focusing on the possible rather than on the difficult. This combination doesn’t leave much room for rigorous self-assessment.
The problem often is these three key issues are glossed over. We tend to think all three are in good shape, and justify these thoughts by superficial considerations and wishful thinking. We also want to believe what we preach to the point of tweaking the facts to fit our needs. Although drinking your own Kool-Aid may be enjoyable, it usually leads to an upset stomach.
The solution is a product management audit—a practical way to counter neglect and false optimism and focus on thoughtful decisions and a justified positive outlook.
The solution: a product management audit
What is a product management audit? First, it’s a review. An analysis of the state of everything related to product management. The goal is to step out of the daily grind and take an unbiased look at issues, and grade the progress achieved towards solving each of them. It has some similarity to a financial audit. It’s a step by step, “by the book,” analysis of everything that impacts product management. It’s a short, intense, and repeated effort.
- What’s the typical framework of a product management audit? For each issue related to product management, one should take a critical look at:
- Facts: What’s known / not-known?
- Confidence: How confident are we to handle problems related to the issue?
- Responsibility: Who’s responsible for resolving each of these problems?
That’s the minimum, but a fundamental one. Of course, there are ways to augment the review by considering objectives, teams, resources, timelines, plans, etc. I would argue against taking it on. Always make sure the basics are sound before diving into the sea of details. Address details as a separate exercise.
Can an audit yield easy-to-understand-and-track metrics? Yes it can! And just two metrics will do. An “average” score (call it the A-score) and the “fix-it-now” score (the F-score). The A-score is a geometric average score of all factors under consideration. And the F-score is the lowest score of all individual factors.
Why is geometric average the right one to use? Think of a car’s tire pressure. If three wheels are at 100% but one is at 0.01%, the arithmetic average (75%) says “it’s still OK to drive” but the geometric average (10%) says “fix the tires now!” Clearly, you need all wheels in good working order to succeed.
The good thing about having numerical scores is they help track progress. They also help identify the bad outlier factors that need fixing immediately.
Example scoring of strategy
Facts: Consider three strategy-related factors that can be scored with the good-bad percentage metric. For example, our strategy statement is 80% completed and articulated (80% good, 20% bad). And, 75% of our external strategy reviewers approve of the risks we’ll be exposed to and bets we have made. And, 20% of our managers understand and accept our strategy. That yields A=49% and F=20%. That’s very bad.
Confidence: We’re 80% sure our strategy will not need to be modified before we ship the product. And, we’re 50% sure our business can survive bad outcomes of our bets. And, we’re 100% sure we will not run into any technology showstoppers. A=74%, F=50%.
That’s not too bad but not good enough.
Responsibility: For 10% of our still-open strategy issues we have an individual identified who’s responsible for closing them. A=F=10%. Really bad!
Total score for Strategy in this example: A=47%, F=10%.
These are dismal results, but the good thing is: they clearly point to areas where improvement is needed.
Example scoring of Target Market
Facts: We know user needs of 80% of the target market segment. We have articulated competitive win scenarios for 20% of anticipated competitive bid types. 100% of our identified target adopters are committed to do beta testing of our product. A=54%, F=20%.
Confidence: We are 90% confident that our value proposition is compelling to our target customers. We are 80% confident customer needs will not change before we ship the product. We are absolutely (100%) sure of our target market size. A=90%, F=80%. Excellent result—and therefore solid scrutiny is needed to verify this score.
Responsibility: We have individual owners for 50% of win scenarios that are still missing. A=F=50%
Total score for Market: A=67%, F=20%. Better than Strategy, but still far, far below target.
Example scoring of Product Requirements
Facts: The Product Requirements Document (PRD) is 100% completed and reviewed. 90% of individual requirements have defined priorities associated with them. The review found 80% of these requirements are clear to reviewers. A=90%, F=80%. Good.
Confidence: We are 60% confident that requirements define all factors critical for winning against competition (Does this agree with the estimate for competitive wins in the Market section? What if it doesn’t?). We are 90% confident that priority-1 requirements must be satisfied before the product ships (Which ones are the questionable 10%?). A=73%, F=60%. Still needs fixing.
Responsibility: We have one owner for 100% of the requirements that need better prioritization and/or clarification. For 20% of items that need clarification, we have identified reviewers that will decide if they have been sufficiently clarified. A=45%, F=20%. Bad.
Total score for Requirements: A=69%, F=20%. Well, this is embarrassing.
And finally, aggregating all scores, the total Audit score for Product Management in this example is: A=60%, F=10%.
I wouldn’t invest in this example startup until these numbers are much, much higher. Can they get better? Of course they can. Once you know—as illustrated by the audit results—what needs fixing, it’s easier to focus your energy and improve the situation.
Audit template
Whether you decide to do it or arrange for an independent audit, it’s good to have a solid understanding of how it works. There are three key aspects of an audit: First and foremost, it’s the scope: the list of questions or factors to examine and review. Second, it’s the mechanics of organizing results and computing metrics. And finally, but perhaps most importantly, it’s the veracity and validity of the audit. A template shown on the next page may be of assistance for the first two. As for the third, it’s a matter of trust in the person(s) doing the audit.
Questions can be many and varied, but not all of them are the “right” questions for a product management audit. What are the right questions? They are largely defined by the intersection of the two sets of topics discussed earlier: “strategy-market-requirements” and “facts-confidence-responsibility.” The ones in the preceding section could be a good starting point. How many questions to ask? As few as possible that are the most meaningful in your situation and answerable with a numerical (percentage) metric.
Some questions can be answered with a precise (countable) number. For example: “What fraction of individual requirements in the PRD have priority assigned to them?” Others can’t. For example: “What is our level of confidence in factors that will assure our wins in competitive bids?” Still, a rough percentage estimate can be defined and be useful. Just make sure the audit captures and considers not only the estimate but also its source.
- Going beyond the audit questions used in the example above, other fact-related questions may include:
- What fraction of our value proposition is uniquely differentiated?
- What fraction of our efforts brings a distinctly new value to the market?
- What fraction of our target market could materially benefit from the full set of features of our product?
- What fraction of the cost of the solution implementation by our customer will be due to the purchase of our product?
- What fraction of our product requirements directly map to our value proposition?
- What fraction of our product requirements represents new and differentiated value?
All of these questions are the “right” questions for the audit. That’s because for each of these questions, a high percentage-score answer represents a “good” answer. The higher the score, the better.
Mechanics of the audit can be supported by a spreadsheet, like the one shown below. The auditor can complete the “Questions” fields and associated “%” estimates, as the audit proceeds.
The example above and the template need a fair amount of specificity. The main thing about the Audit is doing it, applying serious qualitative analysis and using the score as a tool, first for critical thinking, then for tracking progress—until the next audit.
Who needs (and wants) the Audit?
Certainly top executives in the startup should want it, as should the board and investors. If there is one, so should the product manager. Best startups may not need product management audits—all’s working well there. But the vast majority of startups would certainly benefit. What if a company does not yet have a dedicated product manager? Often, the founder drives early product definition and its creation. However, the founder does not usually have the time and inclination which often leads to unjustified optimism about the product in the market and vague product requirements. An audit can really help.
Can it possibly happen that top executives of a company would not want a product management audit? Entrepreneurs are typically strong willed and may simply find it difficult to deal emotionally with someone else “checking” the validity of their own vision and perception. Weak entrepreneurs may even fear less-than-stellar results of an audit may negatively impact investors’ view of the company. It is possible that some companies would prefer not to do such audits. But, the value of an audit is not in finding fault, it is in confirming that everything is fine and if not, in identifying areas that need attention. The audit results present an independent opinion, a tool facilitating good decision making for product management purposes.
You can do an internal audit but it’s best to rely on someone external. Just think about a due diligence process a startup goes through prior to a round of funding. Would an investor ask the company itself to perform such due diligence? Would you rely on the analysis for fund-raising purposes to actually drive your business? The #1 objective of that analysis was to raise funds and that calls for a level of optimism that may be dangerous in day-to-day business operations.
Many startup leaders rely on their network of friendly, non-competing, experienced (read: successful) executives for sanity checks on their business. That’s good, and necessary. But it’s not sufficient. First, it’s always broad-stroke. Second, it only involves the top layer of the company. The net result is similar to the frequent result of offsite executive meetings: Executives have differing recollections of what was decided and few non-executive people in the company know what those decisions were.
This article focuses on product management in early stage companies. However, product management audit can also serve well in established companies, especially when they work to develop a new product for their portfolio.
So, if you decide you want a PM audit, who should do it? Do it yourself-—if you can trust yourself in terms of experience and motivation—but it is best to rely on the objective judgment of a trusted, experienced, and independent professional.
Summary
Effective product management requires periodic rigorous review of key issues affecting the product. An audit of product management can serve a company well when used as a practical tool to evaluate and track progress and identify weak links. The audit, to be objective, is best done by someone independent, someone who’s not involved in the day-to-day workings of the company.
The scope and details of an audit must be adjusted to the needs of the particular company. The main point is just doing an objective audit. There’s much to be gained from going through it and applying the lessons learned in the company’s drive to product success.
