Earlier last week I was sharing with Crista the process I use to efficiently work through survey feedback, which reminded me of a quote from Matt Mochary,
If you find yourself explaining something more than once, itâs worth writing down as youâll probably need to explain it again.
Matt Mochary, The Great CEO Within
So hence this blog â I hope this will serve as a reflection that this is by no means the only or necessarily the right way to do it, I just wasnât satisfied with the current process.
The problem with the process that Iâm trying to solve is when a product is still in the MVP stage and Iâve aggressively acquired a variety of users via every organic/DTC method in the book â Iâve got everyone from my aunt and uncle to my postie and his dog using the product. So how do I find out whoâs feedback I should prioritise? Who is actually best suited for my product? What market is my product best fit for?
This problem is further exacerbated as the current process also falls victim to information overload. We want as much feedback as possible, but once it has come in we donât know where to start unpacking it, and even worse when it comes to sharing it back we overshare with no overarching narrative to the information.
This method I describe below is how I efficiently work through survey feedback and craft a story that best highlights key information:
- Trace the lines I want to cut â Asking the right questions
- Cut to find the outliers â Who loves our product and why?
- Have an opinion â Articulating our observations
- Tell your story, own the narrative â Effectively communicating back to stakeholders
Trace the lines I want to cut
No survey will be perfect or give better insight than talking to customers, but I believe this method can be used to refine how we approach feedback analysis.
The most important step before sending out a feedback survey is to make a note of the specific knowledge that it will provide, think about the purpose of the product, what success metrics or hypotheses do I have, and what do I need to quantify to help me to understand if I've achieved them?
The most common metric we use at Startmate is NPS. (Net Promoter Score). In short the purpose of NPS is to see what the likelihood is that an individual would be a promoter of a program or product. For context, at Startmate we aim for a program outside of MVP stage to be pushing for a NPS of 75+.
NPS by itself is only really good for external stakeholders and leaders to get a snapshot of how a program or product is tracking. The problem is that this doesnât tell me what is going on or why, which means we need to add additional questions to help refine our analysis process.
Measure twice, cut once.
Every survey should have at least 1-3 categorical questions that will allow us to cut and segment information to help decipher the feedback. The questions used should inform the answers to our success criteria.
Ideally before I onboard users, I would ask them categorical questions in their onboarding that I can tag to their feedback to give me more context and as an added benefit, let me ask fewer questions in the final feedback survey for a better response conversion rate and instead focus on questions that let me benchmark a userâs progress over a timeframe.
Disclaimer: These categorical questions are additional to the typical feedback questions we would ask in our survey
One categorical question should be related to a userâs product experience. NPS is a good example of a categorical product experience question, although we ask people on a scale of 1-10, there are really only 3 categories, which gives us a good place to start segmenting the data later on:
- đ Promoter (9-10)
- đ Passives (7-8)
- đ Detractor (0-6)
Another question that we experimented with for the MVP of the Startmate Founder's Fellowship, alongside NPS, was Rahul Vohraâs method that they used at Superhuman. Where we instead asked:
âHow would you feel if you could no longer use the product?â
- Very Disappointed
- Somewhat Disappointed
- Not Disappointed
-Rahul Vohra, How Superhuman Built an Engine to Find Product Market Fit
The reason I like this question is in the framing â instead of using a push question such as NPS, where we ask the user if they will promote our product (a question that is fundamentally idealistic and would fail the Mom Test), this method frames the question as a pull by metaphorically pulling the rug out from under a user to see how they would feel, and gauging that response
Itâs not anyone elseâs responsibility to show us the truth. Itâs our responsibility to find it. We do that by asking good questions.
-Rob Fitzpatrick, The Mom Test
Cut to find the outliers
Segment to find your supporters and paint a picture of your high-expectation customers.
-Rahul Vohra, How Superhuman Built an Engine to Find Product Market Fit
Weâve traced our lines, the survey has been sent out and the feedback has come in, time to start cutting! The first place to start is with those categorical questions above and group users by their shared experience score. ie Promoter, passives, and negatives.
Start by measuring the percent who were a promoter (gave an NPS of 9-10) or if youâre using Rahulâs method, those who selected âVery Disappointed.â Rahul suggests that companies that struggled to find growth almost always had less than 40% of users categorised as a promoter or responded âvery disappointed.â If all has gone well with the user test weâll see a healthy percentage of +40% users in that promoter group, so then we can shift our focus to figuring out how do we get the rest of the fence-sitters (passives) into the promoter camp? This is where qualitative feedback becomes important.
What type(s) of people love the product?
Letâs have a more thorough look into our promoter group, what type(s) of people love the product? This should be relatively straightforward but here are some good starting questions:
- What occupation do they work in?
- What industry is their company?
- What was their usage of the product/platform? - assuming you have relevant data for this
- What stage was the user in before and after using the product? This can be an indicator of a group of users who had experienced what is called the Hero Journey whilst using the product
- Age, gender, etc, there are countless examples
If there are significant groups in this segment, make note of these outliers as they will come in handy later. There's also a good chance there won't be any significant groups, don't worry as this is something we can optimise the user groups for in future product tests and surveys.
and, why do they love the product?
We now want to look through that promotor group and read the qualitative responses to our questions, as mentioned above this will differ from product to product. The overarching goal is to understand why promoters love the product. This can be one of the more manual parts of this process, but to avoid being overwhelmed I like to use a tally system.
I will put the survey question that is being answered at the top of the page and then proceed to read each response, as Iâm reading I started bullet-pointing themes that show up, and as I read more feedback that is related to that theme I add a tally. For example:
What would make the product better for you?â
- Faster support communication - IIIII
- Clarity of onboarding instructions - II
Quite often Iâll read a response and will be unsure if it fits an existing theme, in this case, I will make a new theme for this point, and later on, I can always concatenate it to another theme later when more context is relevant, or make it a sub-theme under an existing one.
Once Iâve completed this process, I add my tallies and rank them from most to least, but before I start making any conclusions I need to now go through the passive group and repeat this process.
Passive users give better insight into our product than promoters.
This may be a controversial opinion but I strongly believe without the contrast of haters and fence-sitters to our promoters and evangelists, we will never truly understand what makes our product great. In the wise words of Ted Mosby,
Every night canât be legendary. If all nights are legendary, no nights are legendary.â
-Ted Mosby, How I Met Your Mother
From going through our promoter groupâs feedback we have already benchmarked what users love about the product, so now by following the same process of analysing the what and why of our passive user group we will be comparing the data against the promoter group to see what made them rate their experience differently which ideally will lead us to our best product wins.
Donât be alarmed if, for the most part, we donât see anything too different, most people will say the same feedback but they may have a stronger deposition to being pessimistic in the way they respond to questions (which isnât helped if youâre akin to having a negativity bias). What we want to look out for are your outliers, groupings of feedback that are significantly different from our promoter group.
Significant differences are subjective and will vary from product to product but someone whoâs spent enough time on a product should know when they see themes emerging that contrast the promoter group. I suggest working through the questions below to get the olâ noggin working
- What areas did the passive group have a significantly different experience to the promoter group?
- and why did they have a different experience?
- What would push someone off the fence (passive) to a promoter?
Hopefully, by this point, conclusions and theories are probably starting to form, make note of them under each group of feedback to deal with later, we want to work through this efficiently and keep going through each category.
By the end, if we havenât found any significant experience differences between the two groups, I suggest trying to group the passives and promoters differently, to see if you find something significant, if that fails you may not have enough user feedback or you need to change the persona definitions of users, which usually requires more information from them.
What do I do with my detractor user group?
Some will advise that you should politely disregard those who would be a detractor of your product. They are so far from loving you that they are essentially a lost cause. I agree with this in principle but would say that I prefer to put this group to the side till after I've finished my analysis, I donât want to be ignorant to a potentially poor user experience, especially when you're unable to see the user in the digital world.
Have an opinion
Weâve cut our feedback into groups, understood what users we have and why they love our product. We probably have a few theories at this point based on the significant statistical differences weâve observed between our promoter and passive users groups. We can now turn these into informed opinions and actions that we want to implement to improve the product.
Start by articulating your observation as a statement, for instance:
The level of founder connection and strength of their explorer group was the most important deciding factor to a fellow rating their experience highly
Now using your data observations from the last step, provide evidence to back up this statement. Examples of data that can be used: percentage comparisons, the difference in the number of users who reported an issue.
The level of founder connection and strength of their explorer group was the most important deciding factor to a fellow rating their experience highlyâ
- 17/21 promoters said the connection to other fellows was their favourite part with only 6/21 saying they had an average or worse experience with Explorer groups.
- In comparison, 21/25 passives said connecting with founders was the best part of the fellowship but had a higher number of 15/25 that said they had an average or worse experience with Explorer groups
and lastly, to drive the point home, use quotes to reinforce the readerâs belief in the theory
My Explorer group has been a huge part of why I enjoyed the SFF1. Being surrounded by people in the same stage and mindset is very helpful."
-Promoter respondent
Tell your story, own the narrative
In my last blog, I mentioned that in a decentralised team like Startmate, effectively communicating with team members and leaders is a necessary skill. Unfortunately, a bunch of opinions on a Google Doc wonât get us anywhere, but purposely owning the narrative to tell the story we want to tell is where we give magic to digital words and lead.
Effectively communicating our report is a lot easier when we remember in most instances the reader will not have had first-hand experience working with this set of users so itâs best to craft your report around these questions:
- What impression do we want to convey after reading the report?
- If a reader can only take one thing away from the report what would we want it to be?
- What actions (if any) will we make from this report?
- Optional: How will we keep ourselves accountable?
At the end of the day a fancy report with a bunch of stats and pullout quotes wonât convey the message by itself, only when we stand up and deliver a compelling story will we have a chance at exciting our team and successfully sharing our learnings.