Oftentimes consultants such as myself work with clients who don’t have much experience running surveys.
But from time to time I run into people who are very familiar with doing surveys. So familiar, in fact, that they can get too close to their projects to see potential issues.
This case study involves just such a client.
The most important takeaway from these types of projects is that even if you have experience designing surveys, you can still benefit greatly from having an impartial third party review your survey and offer suggestions.
Wildflower Marketing (real name changed) uses surveys frequently to test marketing ideas, profile target markets and evaluate new product ideas. They usually come to me with a draft survey and ask me to review and comment on their questions.
This case study is a recent request from this client.
I’ll go through a sample of the input I gave on their survey questions. Along the way I will highlight the general principles that specific examples help illustrate.
Survey Questions From Marketing
I was recently sent a draft of a questionnaire from Wildflower Marketing and asked to comment.
My first question back to the client was, “What is the purpose of your survey? What are your main objectives?”
General Take-away #1: One can’t make meaningful comments about a survey unless he or she knows exactly what the surveyor plans to do with the information.
In this case, the main purpose of the survey was to obtain information about the target market, herbal tea drinkers, in order to help establish marketing strategies for selling herbal tea.
The survey’s first question was, “What do you consider to be herbal tea?” which had multiple-choice selections.
The second question was, “Do you drink herbal tea?” I recommended that in between question one and question two my client include a definition of “herbal tea” for all respondents to use for the remainder of the survey.
General Take-away #2: Make sure terms and jargon are defined unambiguously. The reliability of data produced by a survey question depends on a shared understanding of all relevant terms.
As the survey went on I had additional feedback for this experienced survey creator. Examples of their questions and my suggestions follow
Making Multiple Questions Relevant
Q. Do you MAINLY use herbal tea bags or loose tea? * [Select one]
- Tea bags
- Loose tea
- Both (add this option)
- I don’t know
This simple change makes it easier for the respondent to answer and actually provides more information. No evaluation of “mainly” by the respondent is needed, and information about how many use both methods is added.
Below is an example where the use of a similar type of word, “typically,” can actually make it easier to answer for the respondent because it encourages the respondent to go with their gut feeling.
Q. When you drink herbal tea, how do you typically prepare your herbal tea?
[Select up to two]
- Iced
- Hot in teapot
- Hot in cup
- Sun tea
- Other (please specify)
Survey Question Choices and Bias
Q. What is your MAIN reason for drinking herbal tea?
[Assign no more than 100 points in total across the categories below]
- Flavor
- Health benefit (e.g. digestive, detox, laxative, cold care)
- Comfort (e.g. soothing)
- Habit
- Morning ritual
- For a “pick me up”
- Boredom
- To warm up
- To be sociable
- Instead of coffee
- Doctor’s orders
- Don’t know
My client has his own ideas about why people drink tea, but it doesn’t take much thought to think of possible other reasons.
Above, you can see the items I added as suggestions in bold.
If a respondent sees his thoughts in the question options, it makes it easier for him or her to answer and it produces better data.
General Take-away #3: Don’t bias your survey questions toward your specific objectives. Work to expand your thinking toward how a respondent might think. You can also accomplish the same thing by having someone else review your survey and provide his or her input.
Limits on Required Survey Questions
At this point in my review I noticed all of the questions on the survey were required. At best this is totally unnecessary, and at worst it is extremely annoying for the respondent.
General Take-away #4: Only make a question required if absolutely necessary like when it is a branching or skipping question.
Understanding Why You’re Asking Survey Questions
The next question on the survey was, “How often do you buy herbal tea?”
This question seems harmless enough, but I asked how this information would be used.
It would be impossible to correlate the answers with the amount of tea one drinks or how much tea they buy or even how often they drink tea since there is no mention of how much they buy or when they buy.
I encouraged my client to more fully explore the reasons behind this question and what he was really looking for.
Here were the next couple of questions:
Q. Have you ever had any of the following health conditions? [Select all that apply]
- Anxiety disorders
- High blood pressure
- Back Pain
- Cancer
- Sexual conditions
- Migraines
- Irritable bowel syndrome
- Diarrhea
- Obesity
- Depression
- Skin Problems
- . . .
- Etc. (the list goes on!)
- None
Q. Which alternative medicines or therapies have you used? [Select all that apply]
- Yoga
- Massage
- Homeopathy
- Meditation
- Martial arts
- Acupuncture
- . . .
- Etc.
- None
At this point in the survey my questions were: Are these questions necessary? Where did the lists come from? Where are you going with these questions?
These last two questions hit me as coming out of the blue.
If these questions were necessary, then it would be important to prepare the respondent for them and let the respondent know where you are going with this data.
Something as simple as, “Some people use herbal teas for medicinal purposes, have you considered their use for any of the following conditions? [Select all that apply]” could work.
General Take-away #5: Think about how your questions are going to make your respondent feel.
In general people will answer personal questions or questions about sensitive topics but they need to feel it is both necessary and critical information for you, the surveyor, to have.
Throwing in a question totally out of the blue (of a personal nature or not) can be a turnoff, especially when the purpose for the question has not been made clear.
Guidelines for Using Ranking Questions
My next recommendation was for my client to change their ranking question:
What are your TOP criteria for selecting a brand of herbal tea? [Place in rank order of importance]
- Clinically tested
- Price per container
- Environmental practices of manufacturer
- Price per bag
- Flavor
- Certified organic
- Effectiveness
- Fair trade certified
I prefer to use checkbox questions as opposed to the ranking question because I feel that ranking questions can be time consuming for respondents.
I recommended that the client change this question to:
Q. What are your TOP criteria for selecting a brand of herbal tea? [Select no more than 3]
This will give them a natural ranking of the items and not increase the burden on the respondent.
General Take-away #6: Ranking questions can be time consuming for a respondent to answer. A good alternative is to ask them to select NO MORE THAN their top X items, where X is roughly one-third of the total number of items.
The Danger of Leading Open Text Questions
My final recommendation was for the client to eliminate the following question appearing at the end of the survey.
Q. Please share any additional information you feel would be relevant to this study.
To me you might as well reword this question as, “I don’t have time to think about whether I’ve included everything. Can you help me out and tell me what might be missing, even though you don’t really know what my objectives are or how I plan to use the information you provide?”
Not a good question to ask.
General Take-away #7: Don’t burden your respondent with work you are unwilling to do yourself. They typically do not have as much invested in your survey as you do.
Designing Better Survey Questions Everytime
Even when you have a lot of experience creating survey questions you can still introduce bias or create fatiguing question types for your respondents.
Having someone else – whether they’re a survey expert or not – review your questions with each survey is a great way to make sure your question wording is staying on target.