Integrating Users into Product Development
A new product rises from the creative imagination and technical expertise of a person or a team. The product may arise from an observed need, a new technology, or a new application of an existing technology. Many times it will be created in some sort of isolation—perhaps in the fabled garage or maybe in a team that has been given the independence to create something new.
More often then not, the product’s intended users are conspicuously absent in the creation process. While leaving the user perspective out of the initial design process can allow for unfettered creativity, it can lead to disaster when it comes time to create a business around the product. From the earliest possible moment, it is necessary to understand if the product’s intended users actually find the product useful enough to spend money on it.
This article will examine how and when to bring the user into the product development process. It will first discuss the kinds of information user testing needs to obtain and at what point user testing needs to be integrated into the development process. Finally, the article will examine what user testing looks like during different phases of product development.
Useful and usable
“In the end, products and software that were faithful to meeting the needs of those that would use it will be useful. And in order to be useful, they will have to be usable so as not to impede their usefulness.”
- J.O. Wobbrock Usable vs Useful
When talking about user testing or user experience, the most common assumption is that the product will be tested to make it more usable for its intended users. However, user testing also seeks to answer another important question, namely whether a product is actually useful to its intended market.
A product’s usefulness determines its value to customers. A product’s usability determines how easily customers can utilize the value of the product.
It is possible to create an eminently usable product that fails because it is of little use to its users, and conversely it is possible to create a very useful product that doesn’t succeed because it is too hard to use. To succeed, user testing must test both the useful and the usable. Depending on the phase of development, the focus of user testing changes between useful and usable.
Building users into the process
“Seeking customer input and feedback is a vital and ongoing activity throughout development, both to ensure that the product is right and also to speed development towards a correctly defined target.”
- Robert G. Cooper Winning at New Products, Second Edition
Another important point is that building products around the user experience means that user testing must be built in as part of the product development process. In other words, it needs to consist of more than one or two quick hits of internal or external usability testing. User-oriented development means that testing with users needs to be as much a part of the process as debugging code is in software development. Doing smaller, quicker and somewhat less formal testing consistently at all key points of the process will pay much greater dividends than a bigger, more formal one time effort.
The main benefit of properly integrating user testing into development is that it provides the project team with consistent feedback on whether the product is hitting the target. This allows the team to avoid unnecessary rework, and deliver the right product on time and on budget.
The product development process has been divided into many different phases and stages by many different authors. However, for the sake of this article, I am going to divide the process into three main phases: Concept, Development and Release/Life Cycle. The Concept phase contains the genesis of the product, when the idea for the product begins to take its initial shape. Development is the process of taking the initial concept from the whiteboard and building it out so that it is ready for release. Release and lifecycle pertains to the product’s release to market and subsequent revisions and releases. The more the user experience is considered in each of these phases, the better chance the product will achieve success.
Concept Divining user needs and wants
“Whenever you gather requirements, the challenge is to gather the real requirements. Sometimes discovering the real requirements calls for digging beyond the surface requirements.”
- Steve McConnell, Rapid Development
During the concept phase, it is crucial that a product’s usefulness be tested and understood. Without a fundamental understanding of the value of a product for its intended users, product development is built on assumptions that amount to educated guesses, at best. Very rarely do the assumptions of product developers match the actual needs of the intended users, and without a strong grasp of those needs the best efforts of the Development team will most likely be spent on a product that users will not embrace.
So, how do you find out if something is useful to people? The most obvious and direct way is simply to ask them, and certainly this is an important first step in understanding user needs. There are several techniques to directly gather requirements from users: conducting focus groups, conducting surveys, and, of course, directly interviewing individuals.
Talking with people in order to understand their work and business problems can elicit good information, but it can also generate a puzzling list of requirements that can sometimes be contradictory and misleading. This is due to the fact that, by and large, people are consumed with their day-to-day tasks and cannot often verbalize or imagine a solution that would truly make their lives easier.
The most powerful technique for understanding user needs and desires is carefully observing them in their environment and participating in their world. This kind of user research is called ethnography, and it is especially adept at providing context and clarity to user requirements. By understanding business tasks from the user point of view, and looking for patterns and opportunities to significantly improve customers’ lives, ethnography is very effective in providing developers the necessary information to create useful products.
Many useful and successful products had their genesis in ethnographic techniques: VisiCalc®, the original spreadsheet, WordPerfect®, one of the very first word processors, and the Zip® drive were created by product developers observing people’s work practices, and then using their creative and technical expertise to craft innovative solutions.
Development Keeping users in the loop
“Maintain regular customer contact to keep yourselves moving forward.”
- Hugh Beyer & Karen Holtzblatt Contextual Design
Taking a concept or initial design from the whiteboard to release is an iterative process—the product evolves over successive rounds of development marked by specific milestones. As each iteration evolves, it is important to keep users in the loop to ensure the product is meeting or surpassing expectations. Successive rounds of testing will ferret out issues and problems as they arise and keep development surprises—which generally lead to expensive rework—to a minimum. User testing during development evolves as the product moves from concept to release. As development progresses, user testing begins to focus on making the product usable as well as useful.
In the early stages, as the team is assimilating the information from the concept phase testing, a good approach is to conduct user testing with simple paper prototypes that convey the intended functionality and structure of the product. Paper prototypes allow users to impact a design at the most useful time—the early stages of development before the design has been locked down. Testing with paper give both the users and the project team a sense of freedom to change and experiment that is not possible further down the road. Many times major design flaws and decisions can be addressed through initial rounds of testing with paper prototypes.
As development continues and successive iterations of the product become more sophisticated, consistent user testing will uncover performance issues and usability issues as they arise, as well as ensure that the design is providing the intended functionality.
Frequent usability testing with functional prototypes will provide a steady stream of feedback to the project team. This feedback acts as the final arbiter on the many design decisions that need to be made as the product progresses, and serves to keep the project team focused on the target. If all goes well, the end result is a product that customers find both highly useful and extremely usable.
Release and life cycle
Once a product is released, essentially the process starts all over again. The big difference is that user information will funnel back to the product team through avenues other than direct user testing. Support calls and logs, feature requests, and information gathered by the sales force provide a great deal of data to work with as the product team begins the task of creating the next release. If this data is properly harnessed, it can provide a road map for the project team developing the next release.
While the user data may have different sources, the process of building the next revision of a product takes the same course as building the first version. The process starts by focusing on user needs and desires, understanding user workflow and obstacles to develop a set of requirements. Again it may be necessary and desirable to spend time observing users with the product to place feature requests in context.
As the release moves through successive iterations from initial design to release, usability testing with increasingly sophisticated prototypes will again allow the project team to ensure that the delivered product will hit the target.
By integrating user-testing into each phase of the product development process, companies can better ensure that they create a product that customers find both useful and usable. In the end, that is the only sure way to deliver a product that people will want and, more importantly, a product that people will buy.
Unfortunately, we don’t live in an ideal world. The pressures of time and budget, of getting a product to market as rapidly as possible, as well as other work factors make it challenging to implement user testing in an ideal fashion. What follows is some practical advice on integrating user testing into your product development process.
Focus on business questions
By its nature, integrating users into the development process forces a team or company to stay focused on the main business question: how do we deliver a product that people will buy? At each stage of development, user testing provides answers that can guide the team in building a successful product. During the concept and initial design phase, user testing seeks to answer questions about what fundamental user needs the product or technology can fill, what features and functionality customers desire most, and what opportunities exist in the market. As development proceeds, user testing seeks to answer questions regarding how to organize features and develop an interface that mates the product to the users’ workflow. After release, user-testing focuses on specific product improvement goals, such as reducing support calls and their attendant cost, as well as keeping pace with technological advances and user needs. Setting goals and keeping an eye on what it means for the business can help prioritize and determine testing needs.
Budgeting for user testing— how much is enough?
So, how much of a product development budget should be allocated to user testing? The usability gurus at Nielson Norman Group conducted research to determine if it was possible to determine a rule of thumb regarding how much of a project budget should be devoted to usability testing. They surveyed 863 web and intranet projects that involved systematic usability testing, and found that usability testing costs were between 8% and 13% of the project budget, leading them to suggest 10% of a total product budget as the sweet spot in terms of return on investment. (Nielsen, J, Alertbox: Return on Investment for Usability, January 7, 2003)
Of course, every project is different and depending on the kind of product and the scale and scope of the project, user-testing budgets can vary. While the 10% figure represents just one study’s perspective on user research budgeting, it does offer a sense of user testing priority in relation to the entire development budget.
One of the biggest issues with user testing is getting it to happen in the first place. The twin pressures of time and money are usually the most commonly cited reasons for not conducting testing, especially in the current economic environment. However, as we saw in the first part of this series, user testing generally pays for itself both in time and in money by keeping the project team focused and identifying design issues as they come up, not to mention delivering a product more likely to be accepted by customers. There are numerous objective case studies that demonstrate that in the end, it is much more costly to rebuild a product that customers reject on the first go around than to take the time and money to get it right the first time. Presenting decision makers with real-world case studies and strong financial reasons to implement user testing provides the most compelling case.
It is crucial that the main decision makers on the project team and those managing the budget of the project team see the value of systematic user testing. Without management buy-in, it is not likely that users will be part of the development process and if some user testing is conducted, the feedback will not be valued.
Do what you can on your own
Since user testing works best when it becomes a systematic part of the development process, it may make sense to develop the ability to conduct user testing in-house. There are many things teams and companies can do on their own to understand their customers. The first place to start is to maximize the customer information that already exists within the company walls. Support calls, feature requests, sales force feedback and web logs can offer tremendous information about customers and how they relate to the product. When collated and parsed, these disparate pieces of information can yield important insights into customer needs and desires, and identify specific goals for the product team to address (i.e. reducing support calls around a specific feature or installation).
Beyond the information gathered from internal sources, it becomes necessary to solicit feedback from users in a more direct fashion. Using the information from internal sources, project teams can understand and segment the different kinds of users working with the product and make decisions about what kind of testing needs to be done. The kind of information sought determines the type of testing. Broadly, user testing breaks down in to qualitative testing, which focuses on understanding in-depth the experience of a small number of users, and quantitative, which seeks to understand market issues with a larger number of statistically representative users. Qualitative testing provides answers to questions regarding user experience. Qualitative techniques include usability testing, focus groups, one-on-one interviews and ethnographic research. Quantitative techniques focus on the development and analysis of surveys, and seek to answer questions about market size and opportunity.
There are several options for both qualitative and quantitative do-it-yourself user testing. For usability testing, you can set up a lab on-site where you can invite users to work with your product. If space is not available on-site, there are off-site usability labs that can be rented. Another option to consider is creating a panel of users, ideally representing your target market, to quickly evaluate new features and test prototypes. Users can be incented with discounts on products, priority support or other kinds of compensation.
For quantitative testing, there are websites that allow you to set up your own surveys and email them to a list of users that you provide or to a sample provided by the site. The sites provide templates on which to build your own surveys. Visit Zoomerang at www.zoomerang.com to see an example of a do-it-yourself survey site.
While all of the above is possible to do in-house, it is unrealistic to expect good information from in-house user testing without undergoing at least some basic training on how to recruit the right users, conduct usability tests without asking leading questions, and accurately analyze test information. There are user research consultants who will train teams in usability testing techniques. Several local universities also offer extension courses on the topic of user testing (often they are coupled with interface design classes).
Working with consultants
While there is a lot that you can do on your own, there is an art and a science to user testing. As with any skill, the experience and training of the people conducting the testing can have a big impact on the quality of the results. Using a professional usability consultant can dramatically improve the quality of the testing, as well as help get the most out of the resources available. Bringing in someone outside the company also provides a degree of objectivity that is simply impossible when doing testing internally. Many consultants will also manage the recruiting process—which can be time consuming and is extremely important in getting the right people to test. An outside hand can also lighten the load on the project team. In choosing a user testing or usability consultant, it is important to look for a research background, especially if you are looking for help creating the test methodology. Organizations such as San Francisco Bay Area Computer Human Interface Group (BayCHI) and the Usability Professionals Association (UPA) can help you track down a usability consultant. Nearly as important as finding qualified consultants, is finding someone who you can work with, who integrates easily with the project team. Interview any potential candidates to understand how they work, and if their personal style gels with you and the team.
Rates for user testing vary by project and by vendor. Remember, however, that most vendors’ price for a given round of testing is somewhat negotiable. You can work with the consultant to get the most for your money. For example, you can ask the vendor if they are willing to give less formal reports or test with the minimum number of users. Basically, it’s best if you have in your mind what your ideal situation is and then find the consultant who will match you in terms of service and price.
Believe your users
In the course of developing a product, the team and the company become immersed in the products inner workings. The product team and the company make assumptions based on this intimacy, and one of the most important things user testing does is to check these closely held assumptions with the people who will buy and use the product. Sometimes it can be an uncomfortable process, especially when user testing challenges those basic assumptions. As human beings, we generally find it hard to accept that customers do not see the product the same way we do. One of the toughest challenges with user testing is not rejecting user feedback when we disagree with the message. It is important that user feedback, especially when it challenges preconceived notions, be seriously considered. After all, at the end of the day, it is the customer who pays for the product and the salaries of all the people who build the product.
Case Study: The Zip® Drive
The story of the Zip drive is an excellent example of how melding continual focus on customer needs and desires with a compelling product vision and the ability to quickly deliver a product as promised can produce a smash success.
Before the Zip drive, Iomega® was struggling, producing and selling only a few thousand Bernoulli drives every year. In 1994, then new CEO, Kim Edwards had a vision to create a removable storage drive that could hold up to 100mb of data and would cost $200. He committed the company to making it a reality. Iomega hired an outside design and research company to test Edward’s assumptions and gather information that would inform the design of the drive. Researchers observed people using floppy drives, conducted focus groups and surveyed thousands of users to determine if they were on the right path.
The researchers came back with a number of insights that drove the development of the drive. They indeed found that many users were looking for a personal storage format that would help them transfer larger files and backup their data. Most importantly, the research confirmed that consumers would snap up a 100mb drive that was priced at $200.
The user research also pointed out opportunities to create a more usable product. It turned out that many users were scared that their floppies would disappear forever into their computer because they couldn’t see their disks when they put them in their floppy drives. To address this fear, it was decided that the new drive would have a window. They also observed that men and women worked with disks differently, as many women with long fingernails needed to cock their finger in order to grip the disk, so the new drive would have a notch to facilitate women with long fingernails. The research also revealed that most people considered the data on their drives to be “their stuff,” so Iomega decided to market the drive as a place to stash your stuff, even shipping the drives with labels for media that read “Old Stuff,” “Work Stuff,” and other labels that allowed users to personalize their drive.
Armed with this information, the designers and engineers raced to quickly develop a prototype for the drive. The first prototype featured a top-loading door, inspired by those found on portable CD players. However, before committing the prototype design to production, the drive was tested with users to gauge their responses. Their response was surprisingly negative. Users felt the top-loading door was not only too flimsy to withstand the rigors of everyday use, but also would allow dust and other contaminates easy access to their precious data.
The designers and engineers went back to the drawing board, and the drive was redesigned to accommodate the now familiar front-loading door. The case was designed so that the drive could be oriented vertically or horizontally, and was made of a molded blue plastic that immediately gave the drive a more personal feel. Eleven months after the initial concept, the final version debuted at Comdex in 1995, and was met with rave reviews. With a six month lead over competing drives, the Zip drive was a runaway hit, selling one million drives its first year, elevating Iomega from a niche company making specialty storage products to a big player in the consumer electronics market.
IBM Ease of Use website: www.ibm.com/easy
This site features IBM’s take on user-centered design. It features detailed information about implementing user-centered design, business case studies and interviews. The site is a great starting point with links to conferences, articles and books.
Nielson Norman Group Website: www.nngroup.com / www.useit.com
These sites are the home of Jakob Nielson and Donald Norman, who are considered two of the leading thought leaders in web, software and product usability. The sites contain articles and other reports pertaining to usability, and you can sign up for their Alertbox email newsletter. As thought leaders, their opinions are often on the mark, though sometimes controversial.
Good Experience Newsletter www.goodexperience.com
This e-news newsletter chronicles usability and customer experience foibles and achievements in a light and lively style.
Usability Professionals Association (UPA)
Another organization devoted to promoting usability techniques and concepts. For more information, go to the UPA website at www.upassoc.org.
Hugh Beyer and Karen Holtzblatt’s enlightening book on their method of customer-centered product development.
Looking for the latest in product and data science? Get our articles, webinars and podcasts.