As a survey design expert I see the same mistakes in survey design all the time. In fact, last night our neighbor was over and he had just completed a customer survey at a local restaurant. He was complaining that they had asked a bunch of multiple-choice questions that didn’t apply to him. Even worse, there was no “not applicable” option, and the survey required an answer to every question!
This one simple example alone incorporates the first three of the most common mistakes I see in survey design.
With that in mind, here are five of the ten most common mistakes made when writing surveys.
Keep in mind that while it is nice to know the most common mistakes made, the real value is in understanding how to avoid them. In order to do that it is important to understand why each is a mistake.
5 Common Survey Design Mistakes
1. Having little or no understanding of the target audience
This seems straightforward: how can you write an effective survey if you don’t understand much about who will be completing it? The issue is making a connection with your respondent.
The survey writer should know as much as possible about the attitudes and beliefs of the potential respondent. The wrong wording can offend respondents or just steer them away from what you intended. Often, too much focus is placed on what information the survey writer wants to get back, and not enough on what information respondents can provide.
2. Providing multiple choice lists that are too restrictive
It is always a good idea to include answer options that include “don’t know” or “uncertain,” “not applicable” and “other.”
Respondents become frustrated when they don’t see their response in a multiple-choice list. The idea is to weed out respondents that don’t have a clear opinion from those that do. Otherwise you risk contaminating the good responses.
3. Requiring answers to all questions (online surveys only)
Nothing is more annoying to respondents than having offered their time to complete a survey and then having trouble progressing through the survey. A few skipped responses is not going to change your results – and ultimately you cannot force respondents to answer a question.
If they want, a respondent can just close their browser and forget about your survey.
4. Asking too many open-ended questions
It is good to have comment fields, but too many open-ended questions makes it appear that the survey writer did not want to put in the effort to create easy-to-answer questions focused on survey objectives.
The main purposes for open-ended questions in survey design are to provide respondents an outlet for thoughts and opinions that may otherwise distract them from thinking about the questions asked, and to add richness and understanding to the quantitative results obtained.
5. Using ranking questions incorrectly (or overusing them)
The inexperienced survey writer will often overlook the fact that ranking questions are difficult for a respondent to answer and even more difficult for them to analyze and interpret. They assume that the best way to ask a question of one person is the best way to ask the same question of many people.
If I have only one customer, then I would want that customer to rank their priorities (one to whatever). That all changes, however, when I have to consider the priorities of many customers together. Asking respondents to select their top three priorities (or two or four, etc.) creates a natural ranking when the data is summarized.
In my next post, I’ll finish my list and address the last five most common mistakes made when writing surveys.