How product managers can use AI to get more actionable insights from qualitative data

Today we are talking about using qualitative data to drive our work in product and consequently improve sales.

Joining us is Daniel Erickson, the Founder and CEO of  Viable, an AI analytics tool that enables businesses to instantly access and act on valuable insights from customer feedback, saving them hundreds of hours spent analyzing feedback. Before founding Viable, he held senior leadership roles in engineering, technology, and product.

Summary of some concepts discussed for product managers
[2:25] What is the qualitative data you have found useful for making product management decisions?

When most people think about using qualitative data in product management, they think of surveys, user interviews, or getting reactions to a prototype. There’s a huge wealth of other qualitative data that often gets ignored by product teams because it is so hard to use—for example, customer support tickets, sales call transcripts, social media mentions, interview transcripts, and product reviews. Often somebody on the team is responsible for reading through all that stuff, synthesizing it into insights, and disseminating those insights across the team. This is a very manual process, so few teams decide to do the work.

[4:22] What does that manual process typically look like?

It starts with someone on the product team who says, “We need to know more about what our customers need from us.” Then the product leader goes to some poor associate PdM and asks them to collate all of the data together. This person goes to customer support and asks for raw data or asks what the customers are saying. The customer support person gives a response off the top of their head, which is biased and is not the big picture. Then the PdM or a person on the CX team reads through the data, puts it into Excel, and adds a column for bucketing to tag the data, e.g., “checkout” or “onboarding.” The PdM may do this for several data sets, such as NPS and sales calls. They tag each piece of data, find the biggest issue, synthesize the data, and write a paragraph about the issue. That report goes to the top-level leadership.

This process takes a phenomenal amount of time, from 10-20 hours per week. You get analysis for only 5-20 buckets, and because those buckets are so broad, it’s hard to take action on that one-paragraph summary. Unless you spend hours going through every single data point, you’ll miss some nuance. It’s hard to get the fidelity of information you need to act on it.

We found that artificial intelligence is starting to help companies make better product management decisions. Computers can go through the data in less time and in a more nuanced way.

[12:53] How can we use AI for better qualitative data analysis?

The first text analytics softwares could understand what is in a word cloud and identify parts of speech, but a word cloud doesn’t give you much other than some topics you might want to pay attention to. Sentiment charts also don’t show you how to take action. Over time, we have gotten more sophisticated tools to identify different topics. Now, transformer models allow computers to understand language itself. They’re no longer breaking apart parts of speech. They’re using statistics to predict what was meant. These tools are better at detecting sarcasm and agglomerating different wording about the same topic, for example it could group together “checkout” and “cart.” These tools are much more helpful in analyzing large amounts of text.

Our AI analytics tool Viable provides analysis itself. Instead of just grouping things together and identifying themes, it analyzes these themes in the same way a qualitative analyst would. You interact with it by piping data in and asking questions. It produces reports based on the data. Each report identifies a ton of different themes and outputs a full analysis for each one that includes what people are talking about, why they are talking about it, who is talking about it, and what you should do about it. Instead of pumping out a dozen themes like you would in the manual process, Viable pumps out 120 themes that are each all much more actionable.

[19:11] What examples do you have of using Viable to make better product decisions?

We were working with an ecommerce company that ships custom-printed items to customers. They were receiving a lot of complaints about t-shirts arriving damaged. They piped the customer feedback in the form of support tickets into Viable, and we ran our analysis on it. We found close to 200 different themes and then searched through them to find the ones about t-shirts. We found the problem with the t-shirt quality was with the shipping not the manufacturing. They switched shipping providers. The next month, they ran the same report and the complaints about quality after shipping had decreased by 50%. You can use this data analysis for discovering issues and for validating whether or not your fix worked.

[22:02] How can organizations automate this data analysis into their process?

Eighty percent of data that is collected by companies is unstructured text like support tickets. This is data your company already has. The answers to your questions are already there. You pipe your feedback into one system that is your record for customer feedback. Our system produces weekly reports. We show how your complains have decreased or your compliments have increased, your top feature requests, and your top questions. Those rankings will change over time. It’s an awesome way to have a high-level view of your customer in your customer’s voice. All of our analysis comes with direct quotes from customers. We give recommendations for what you might want to do and raw text from customers.

[26:23] You’ve commented before on the single most important question product managers can ask to determine whether they have nailed product market fit. What is that question?

That question is “How disappointed would you be if you could no longer use this product?” If 40% or more of your respondents say they would be very disappointed, you likely already have product-market fit.

Additionally, I suggest you ask three more questions:

• Who would get the biggest benefit from using this product?
  
• What benefit do you get from using this product?
  
• How can we improve this product for you?
  

Make sure all those are open-ended text questions. You can use the answers to improve, and your score for the first question should start going up. As you understand the benefit you’re providing, you can hone in on that value proposition. As you understand who would be using your product, you can further target your audience down to just the people who are going to get the most benefit from your product.

Action Guide: Put the information Daniel shared into action now. Click here to download the Action Guide.
Useful link:

• Learn more about Viable
  

Innovation Quote

“The best way to predict the future is to create it.” – Alan Kay

Thanks!

Thank you for taking the journey to product mastery and learning with me from the successes and failures of product innovators, managers, and developers. If you enjoyed the discussion, help out a fellow product manager by sharing it using the social media buttons you see below.