Advice for Implementing UX Surveys
It's a part of your user experience
A good rule to remember when designing surveys is: No one wants to answer your survey (unless they're really really mad or you are paying them). This rule is especially true for in-app surveys. People are using your app to do a job, accomplish a task, or waste time, and your need for data to prove to the Powers That Be that you and your team are doing a great job and deserve a raise is interrupting what the user wants to do. So think very carefully before implementing an in-app survey.
Another thing to keep in mind is where this survey sits in your app ecosystem. Is this the only interruption within the user experience, or is this one of many pop-ups and ads asking for newsletter signups, purchases, or links to buy the latest and greatest pet toy? If this will be one of many divergent requests for the user's time and attention, you will probably be better off not doing it.
Keep it short, really short
Knowing that, and you still intend to go down this road, I usually advise clients to keep the survey as short as possible using only two questions. The first should be a rating scale measuring whatever key factor your company wants to excel at, and then the second question should be asking them to explain their previous answer.
For example:
How easy or difficult was it to complete what you wanted to do using this app?
Very easy
Easy
Neutral
Difficult
Very difficult
Can you please tell us why you gave that rating?
[multi-line open end]
That's it. Seriously. Two questions.
Now your company may use Net Promoter Score (NPS) to measure success, and there are many thought pieces for and against NPS measurement systems. I prefer pretty much any other scale question to NPS, but your company may be fully entrenched in using NPS scores, and so that's what you have to use. No problem, replace the first question with the NPS question.
If you have the benefit of not using NPS, you have a plethora of options available to you. Decide the single most crucial factor your app needs users to achieve (ease, quickness, happiness, enlightenment, etc.) and devise a question around it. You can use a 5 point scale such as the question above, or you can use a 7 point scale such as this:
How easy or difficult was it to complete what you wanted to do using this app?
Very easy
Easy
Somewhat easy
Neutral
Somewhat difficult
Difficult
Very difficult
5 point scales are easier for users to answer, but 7 point scales provide a little extra nuance. Both are perfectly fine to use; just remember to have your critical positive factor on one end and the opposite of that factor on the other.
Don't cheat by using one question with a multi-factor answer grid, such as:
An example of a multi-factor answer grid.
A grid such as that is not one question. That is five questions stuffed into a one-question trench coat. You know you're cheating, and your users will know you're cheating.
Find a way to allow users to answer the survey without being taken away to another website. Many survey platforms offer this, but at a cost. If your survey takes users to another website to answer, your survey takes them away from the job they came to your app to do. That's not a good user experience, and this survey is a part of your app's overall experience.
When you are writing the response request, you want to make responding to your survey sound as appealing as possible. Use plain language and make sure you say that the survey is only two short questions long and that their feedback will help improve the app. Make providing an answer as enticing as possible.
Response Rate
Keep in mind that your response rate will be really low even if you follow my advice to the letter. More often than not, users will ignore your request to answer your survey. Of the people who answer your survey, their primary motivation will be because they're unhappy and want to let you know and tell you why.
You can address this by offering an incentive through a service discount or a chance to win a prize. Give them a reason to give you their feedback even when you're doing a fine job.
Analysis
When analyzing the resulting data, use the Top Two Box method. What percentage of people answered positively (Very easy or easy) versus neutral or negatively? If you went with a 7 point scale, add the top three boxes together (Very easy, easy, and somewhat easy). That's all you need to know - what percentage of users have a positive experience using your app?
Let's talk about margin of error. Don't freak out when the Top Two Box score goes down by a few points. Don't celebrate if the score goes up by a few points. Calculate your margin of error using a sample size calculator. Enter the number of people who used your app within the period for the data you're analyzing. That's the population. Then enter how many people answered your survey. That's the sample. Then look at the margin of error. A professionally executed survey will typically have a 5 point margin of error. That means the results could actually be 5 points up or down from the reported results.
The same goes for your in-app survey. Depending on the traffic your app gets, your margin of error could be massive. Calculate the margin of error using the number of people who used your app within a specific time frame. The number of people who answered your survey within that same period is your sample. That should be enough to give you a margin of error.
Don't forget to analyze the open-ended data from the second question as well. People freely telling you what they like and didn't like about your app is research gold. Use it. That data can alert you to unmet needs or minor problems that are on their way to being much more significant.
You can analyze the data manually by putting the data into a spreadsheet and then tagging each entry with keywords to help find common themes between the different responses.
You can also use an AI option to help with analysis if you're looking at more data than you can handle manually. But be warned that AI analysis tools are notoriously inaccurate. Computers can't detect the nuance we humans pick up on instinctively.
I hope that helps you reconsider your quest to build an in-app survey. Best of luck.
