Resources > Articles

The Unintended Consequences of Products that Work Too Well

Post Author
The Unintended Consequences of Products that Work Too Well

The Unintended Consequences of Products that Work Too Well

 

RECOMMENDATION ENGINES ARE POWERFUL. At their core, they are driven by artificial intelligence (AI) and designed specifically to provide a more personalized experience for the end-user. The implication is that a more personalized experience will lead to a better experience—one in which users get more value out of the product.

But what happens when a recommendation engine works too well? What if an ultra-personalized experience is actually a bad thing? As product professionals, do we have an ethical responsibility to ensure that we not only design our products well but also protect our end users?

Guillaume Chaslot, a software engineer and researcher at Université Paris-Est., believes we should, and he draws his belief from his own personal experience.

From 2010 to 2013, Chaslot worked at Google and was tasked to improve YouTube’s recommendation engine, called Up Next. He and his peers worked tirelessly on the algorithm, which was designed to maximize a user’s time on YouTube. After all, the more time spent on YouTube, the more advertisements viewers see. And, like most other ad-supported business models, the more time users spend engaging with the product, the more revenue that is generated.

Since his three-year stint at Google, Chaslot has become openly critical of Up Next. “Having worked on YouTube’s recommendation algorithm, I started investigating and came to the conclusion that the powerful algorithm I helped build plays an active role in the propagation of false information,” he wrote in “How YouTube’s AI Boosts Alternative Facts,” published in 2017 on Medium.

In the piece, Chaslot pointed out how the algorithm has caused conspiracy theories, “alternatives facts” and fake news to circulate around the internet. All it takes is for a user to search one topic that is similar to other video topics and YouTube begins encouraging—even automatically playing—more videos just like it.

But if the user is interested in this type of content, is that a bad thing? YouTube’s recommendation engine caught the attention of journalists from The Guardian and Buzzfeed, among others. In a Buzzfeed article, “We Followed YouTube’s Recommendation Algorithm Down the Rabbit Hole,” staffers conducted their own experiments to see whether their queries would result in recommendations to view questionable content.

In one instance, a search of the term “U.S. House of Representatives” led to a PBS NewsHour video clip. But when the staffer clicked the “Up Next” recommendations, it led to a video featuring an Arizona rancher recalling an incident in which he called the U.S. Border Patrol on an alleged illegal immigrant. That video originally was posted to YouTube by a group that was identified in 2016 as a hate group by the Southern Poverty Law Center.

“Today, recommendation engines are perhaps the biggest threat to societal cohesion on the internet—and, as a result, one of the biggest threats to societal cohesion in the offline world, too,” wrote Renee DiResta for Wired magazine. “The recommendation engines we engage with are broken in ways that have grave consequences: amplified conspiracy theories, gamified news, nonsense infiltrating mainstream discourse, misinformed voters. Recommendation engines have become The Great Polarizer.”

Should a company like Google have a responsibility for Up Next to protect its users from videos that contain content that may be spreading misinformation or is potentially harmful? Apparently, the technology giant believes it should.

##tweet##
In January, just after Buzzfeed released its piece, YouTube published changes intended to improve its recommendations. “We’ll begin reducing recommendations of borderline content and content that could misinform users in harmful ways—such as videos promoting a phony miracle cure for a serious illness, claiming the Earth is flat, or making blatantly false claims about historic events like 9/11,” YouTube staff wrote.

So, if Google is taking this stand, should this give us even more pause as we consider what’s right for our own products?

This is a decision that product professionals—and their companies—must make on their own. The case of Google and YouTube is unique; after all, such a popular and widespread product brings more intense scrutiny by the media and public.

We may not face the same kind of scrutiny with our products, and we may not have journalists openly experimenting with and testing our algorithms to explore the potential effect on users. Still, it’s worth considering how our products are designed and asking some important questions:

  • Are all features, as designed today, improving users’ lives?
  • Is it possible for any features to detract from users’ quality of life?
  • Do the metrics being tracked take “quality of user’s life” into account in any way?
  • Is the company willing to change a product to promote a user’s well-being, even if it may affect the overall business model?

The case of YouTube and Up Next illustrates how it is possible for software products to work perfectly as designed, yet still lead to unintended consequences. It’s often said that, in serving as a product manager, having customer empathy is a necessary trait—perhaps even the most important trait. If we truly believe that, we should at least consider the role we play in the consequences of our products.

Author

Author:

Other Resources in this Series

Most Recent

Magnifying glass replacing the letter o in the word Job
Article

How to Maximize Your Product Job Search 

I am no stranger to the product job search. This journey has taught me a lot about navigating the job market, and that's what I want to share with you.
The image features the term use scenario being revealed underneath a ripped piece of paper
Article

What is a Use Scenario [ +7 Examples]

The purpose of drafting use scenarios is to help your development and design teams to start thinking about solutions. Context is the foundation of innovation, and you’ll be providing a tool that will be the starting point for collaborative and productive meetings.
Article

[Comprehensive Guide] Product Owner vs Product Manager

Learn how to separate the roles of product owner and product manager on Agile teams and uncover some common challenges with confusing these roles. Including a short primer on the Agile revolution.
Article

Use Scenarios are Stories That Provide Context

The problem with today’s user stories is that they aren’t interesting. And they aren’t stories. The solution is use scenarios. It’s a narrative. It explains the problem in the form of a real-life story.
Article

Benefits of Bundle Pricing

Bundle pricing is simply a strategy where services or products are packaged together for one (often reduced) price rather than priced separately. This article covers some benefits of bundle pricing followed by a system for getting started.

OTHER ArticleS

Magnifying glass replacing the letter o in the word Job
Article

How to Maximize Your Product Job Search 

I am no stranger to the product job search. This journey has taught me a lot about navigating the job market, and that's what I want to share with you.
The image features the term use scenario being revealed underneath a ripped piece of paper
Article

What is a Use Scenario [ +7 Examples]

The purpose of drafting use scenarios is to help your development and design teams to start thinking about solutions. Context is the foundation of innovation, and you’ll be providing a tool that will be the starting point for collaborative and productive meetings.

Sign up to stay up to date on the latest industry best practices.

Sign up to received invites to upcoming webinars, updates on our recent podcast episodes and the latest on industry best practices.

Training on Your Schedule

Fill out the form today and our sales team will help you schedule your private Pragmatic training today.

Subscribe

Subscribe

Training on Your Schedule

Fill out the form today and our sales team will help you schedule your private Pragmatic training today.