Our guest blogger today is Kathryn Korostoff of Research Rockstar. Research Rockstar’s team of 10 expert instructors teaches 25+ market research topics in a virtual classroom (no travel required).
Professional market researchers know some great questionnaire design tactics to make sure that their surveys will collect the highest quality data possible.
Here’s one you may not know, but it’s easy to implement even if you have no formal questionnaire design training: avoid over-relying on 5-point rating scales.
You know the ones I mean: “Please indicate your opinion using the 5-point scale below from ‘Strongly agree’ to ‘Strongly disagree.’”
Or how about this one, “Indicate your level of satisfaction using a scale where 1 is ‘Not at all satisfied’ and 5 is ‘Very satisfied.’”
Those are just two of hundred-plus variations of the ubiquitous 5-point rating scale.
Professional researchers know that rating scales tend to be over-used.
The overuse of these scales is understandable. They are very easy to write and yield data that is simple to analyze. But they also have drawbacks.
One Person’s 4 is Another’s 5: The Problem With Rating Scales
Most significantly, rating sales are more subjective than you may initially think. To see how this impacts data quality, consider a popular example of a 5-point rating scale: Amazon reviews! Consider these:
Each rated this drone product a “4”, but the supporting text certainly doesn’t seem equivalent, does it? That’s the problem: one person’s “4”, is another person’s “5”. And how I conceive the difference (or “distance”) between a 4 and a 5 may be very different from how you see it. For some people, a “5” is nearly unattainable. Other people are more generous with their 5s.
From a statistics point of view, that makes data from ratings scales a bit messy. How do I take an average (or “mean”) if my survey takers used the scale differently from one another? Sure, I can calculate the average—but how accurate is it? In some cases, imprecision may be fine. Rating scales are easy for your survey participants, and if the data is “good enough,” fantastic. But there are cases where rating scales are not ideal because of the inconsistency issue. If you remember college statistics, this is the difference between ordinal and interval data, and how they should be treated.
In addition, rating scales don’t always yield clear prioritization. For example, imagine you are designing a questionnaire about car stereos and a key part of it needs to measure customer priorities among a set of 5 possible new features. If someone rates every possible feature at a 5 or 4 star, the results of the survey won’t really be helpful.
This is a classic case where a rating scale is not ideal, even though it may be your first instinct. Instead, consider using a rank order scale where respondents list options in order of importance or use a different, more objective scale instead.
As you gain survey design experience, you will learn that scale choice is a critical part of the process. The choice, wording, and formatting of different scales can have a huge impact on a project’s success. Scales must be precise enough to yield actionable information while being written in a way that encourages honest, candid responses. That’s a lot of work for a very limited number of words.
So, do you suspect that you or your colleagues over-rely on rating scales when designing questionnaires? If so you are not alone. And the good news is this is fixable: there are many other scale types to learn about. Try using constant sum scales, rating scales, or explore even more advanced types.
You can try out different scales while using Alchemer. Alchemer includes a wide range of question types used by professional researchers so that you can avoid over-relying on one rating scales Not yet a customer? Start your trial today at Alchemer.com.
Want to learn more about your scale options? Check out the Questionnaire Design 201 course in the Alchemer Collection at Research Rockstar. Next session starts June 8, 2017, and is taught by a live instructor in a virtual classroom.