Quantitative surveys have a flaw: How do you know that the questions you are asking represent the most important factors to the decision you’re making?
If you don’t ask the right questions, or you skip an important factor it could lead to bad conclusions — or worse no conclusion at all.
Here’s an example, imagine you are building a product satisfaction survey. You want to know which product features contribute the most to consumer satisfaction. Presumably you want to know this so your team will build more of those features and less of the other kind.
So how do you know which features should be measured? Your product has over four hundred features! Should you rely on the product manager’s opinion?
You can assume that he understands the product well — but does he understand the consumer? How do you know if you both miss a key feature in the survey? What if that missed opportunity could make or break your company’s future?
The solution is common sense: Ask a small group of your respondents for their opinion before designing your survey using open-ended questions.
We call this qualitative exploration, but in plain English you can call it doing your homework on a survey project. By using this technique you can discover hidden factors that affect the topic of your survey that otherwise might go on unnoticed.
Qualitative Exploration
Using an exploration phase in your survey design has two main benefits. First, you discover underlying factors that might be missed. This avoids the common mistake many researchers make of relying on their own intuition to create survey questions. Second, you’ll be able to build shorter and more focused surveys becuase you’ve eliminated dead ends from your research even before you begin.
This technique is also very useful if you are designing a survey for a product (or problem) you are not very familiar with. Using a more robust exploration phase you’ll gain an understanding of the issues that impact your study and design a more accurate and actionable survey as a result.
All that’s required is asking several open-ended, qualitative questions to a small sample of your survey takers. The questions are designed to illicit a detailed response and drill into underlying factors so you can build a better survey.
Designing the Questions
Qualitative exploration surveys/interviews should be short and simple. The best are 2-3 open-ended questions that describe a scenario and ask for the opinion or experience of the participant.
We suggest creating several questions to focus a participant’s response rather than a single all-encompassing question.
For example: “What parts of the experience did you enjoy or appreciate?” and “Which parts of the experience did you not enjoy?” are better than “Please describe your experience” (which isn’t really a question).
By breaking the task up into a few smaller questions (though still open ended) you’ll get more specific and oddly more verbose replies. This is because specific questions are less confusing and easier for the respondent to answer.
Don’t forget that good survey design applies to qualitative questions too! Remember to avoid asking leading questions, and avoid suggesting replies.
You want the honest unaided answers to your questions that identify factors in your study that are important but may be unknown to you. If you lead the participant to a particular response it defeats the purpose of the project.
Five Low-Cost Methods to Get Responses
When selecting good survey sample, the participants you select should be selected as randomly as possible to eliminate selection bias. When selecting sample for an exploration project you want to keep the same guidelines in mind, but as you are not drawing concrete conclusions about the population from this part of the study you don’t have to spend as much time randomly selecting sample.
Collecting the data from your study can be done in any number of ways, here are five of our favorites:
-
Phone Interviews – This is the simplest, fastest and most universal way to get to explore your survey topic (assuming you have phone numbers for your respondents).
Just remember that phones are a very personal possession and many people don’t like being called or solicited for information out of the blue.
We recommend emailing customers ahead of time asking them if they are willing to answer a few questions before you call them.
-
Online Research Communities – Many larger organizations (particularly in food and consumer products) have started creating online research communities just for research purposes.
They use low-cost social media tools like Facebook, Ning and SocialEngine to create the communities and maintain them with a small number of highly engaged participants that are offered incentives to answer their questions day and night.
Once created, you basically have an on-demand focus group ready to answer your questions in a moments notice.
The downside to this method is that maintaining a community requires time and continual effort — and it’s not much use if you haven’t set it up ahead of time for the project you need to do today.
-
Use a Mini-Survey – using an online survey is very popular option. You can create a short 3-5 question survey of open text fields and disqualification questions if necessary and field it to a small group of respondents.
The caveat for this method is that you should not invite the respondents who take your qualitative exploration study to take your primary survey afterwards. Also, you only need a few quality respondents, so dole out the survey slowly so you don’t burn through your survey population on your exploration study alone.
If you are using a panel company for your main survey – this is your best option for exploration.
-
Use Existing Feedback Systems – If you are surveying about product or customer service satisfaction you may already have a wealth of data available. Ask your service team for ticket content and feedback forms that have open text fields in them. You may also have past surveys and exploration studies you have done in the past. Reduce, Reuse and Recycle — it’s not just for the environment anymore!
Of course, as with the other methods there is a “gotcha” here too –just because the data is available doesn’t mean it’s appropriate or unbiased.
Customer support transcripts and tickets are a wealth of information, but the people that contact support may not be representative of your entire customer base. Remember to qualify the data before you base decisions on it.
-
Ask the Front Lines – In some cases you simply may not be able to access your customers or target audience directly. Consider interviewing the front-line customer service employees that interact with your audience.
It may not be as great a solution as going direct to the target population, but these folks usually have the best understanding of customer needs and issues in an organization. We recommend getting as close to the end customer as possible for these interviews. Avoid interviewing management becuase they tend to pre-filter their responses.
Incentives
If you’ve ever come to our Boulder trainings, or listened to my webinars, you’ll know that I’m not a huge fan of incentives for surveys.
I feel that surveys should be short enough not to need an incentive and the value of the survey should be so clear to the participant that they are more than willing to help without incentive.
That said, there is an exception to every rule. I do feel that participants in exploration studies and interviews often deserve (and need) something extra. After all, one reason we try not to have open-text questions in our primary surveys is due to the time and cognitive fatigue it causes — and where we are asking a group of people to answer a survey that is entirely constructed of open text essays.
If you decide to offer an incentive to increase engagement just remember, the rules for good incentives still apply. Try not to bias your results with a bad incentive choice.
How Much Data is Enough?
Figuring out when you have enough qualitative data to create your survey is tricky. Unlike in a quantitative survey, you don’t have a handy sample size calculator to tell you the target number of responses.
Ideally, I like to get no less than ten (high quality) interviews for each segment in our survey population.
However, in reality we usually get half that number and feel confident that we’ve identified the primary variables about the issue we are surveying.
Note: If you are luckily enough to have existing data from feedback systems this usually isn’t an issue. Usually the issue is that you have too much data and selecting a random sample of it is more of a problem.
Bringing it Back to the Survey
Once you’ve collected the data from your survey/phone calls/interviews you need to analyze it to look for patterns. This is where qualitative data analysis comes in.
Use your survey tool’s text bucketing feature or export all your responses to excel and start looking for recurring patterns that identify variables that effect your learning objective. Take those variables and use them when creating your survey’s questions and multiple choice options for your survey. You’ll likely also start seeing overt trends in the data at this point — but don’t act prematurely!
The greatest threat to the process at this point is your opinions and intuition. When analyzing open ended responses there is always a chance that the researcher may misinterpret or represent what they survey taker intended to say.
Qualitative data is insightful and useful — but it’s not a statistical representation of your population’s opinions and demographics.
That’s why you do a quantitative survey based on this discovery data. To provide statistical certainty to the directional and qualitative data you’ve collected.
Discussion
* Do you use qualitative research when designing surveys? What methods have worked for you?
* Do you have questions about quantitative vs. qualitative?
* Try it out: brainstorm use cases in your organization for qualitative exploration.