In my last article about survey question wording, I mentioned I would talk about the different methods in a survey for getting a ranking of a list of items.
Obtaining such a ranking has wide applications, from marketing (when trying to decide what product features are most important), to customer satisfaction (when trying to decide what improvements should be worked on first), to employee satisfaction (when trying to decide what is most important part of job satisfaction).
So here we’ll look at some detailed examples of how you could get a ranked list without using a ranking question.
Ranking Survey Questions Defined
Before we dive into some specific case studies, let’s first explore briefly what “ranking” means in the context of a survey.
Clearly, if you are my only customer and I ask you to rank five possible criteria in order of their importance, then I’m done. Your opinion is the only one that matters, and I have all the information I need to evaluate each criterion.
However, suppose I have two customers.
Is it still best to ask each to rank order the five criteria? What if I have three customers or more?
How do I use this data to “rank” my criteria?
When I have one customer or one critical party then ranking gives an unambiguous ordering. That changes when you are looking for a ranking from a group of survey respondents. The result of rank ordering doesn’t translate directly from one to many.
In the above example we could make a case for ranking criterion D number one and Criterion C number two, but beyond that things get pretty murky.
Ranking Survey Question Example: Herbal Tea
Let’s say we want to rank the criteria herbal tea buyers use when buying tea. We want to know the best ordering of the following buying criteria for herbal tea drinkers:
- Clinically tested
- Environmental practices
- Price
- Flavor
- Certified organic
- Effectiveness
- Fair trade certified
We assume that knowing our customers’ priorities will help guide our engineering and marketing efforts.
We have already talked about one way to approach this problem: we can ask each respondent to rank order the seven criteria.
Now let’s look at two other alternatives.
Survey Question Ranking Alternative #1: Top Choices
The first alternative is to ask respondents to select their top two or three criteria out of the list. (A good rule of thumb is to have them limit their selection to about one-third of the total number of items.)
This requires the respondent to do a similar evaluation as he or she would in rank ordering the criteria, but the evaluation is not as extensive. The results yield a natural ranking by looking at the criteria according to how many “votes” each item received.
Survey Question Ranking Alternative #2: Distributing Points Among Choices
The second alternative is to give respondents 100 points to distribute among the seven choices as they please. This option, too, requires a similar evaluation by the respondent, but it provides the respondent additional freedom by allowing him or her the chance to divide up the points among all the items. The only requirement is that the sum of points assigned adds to 100.
To summarize, here are the three choices to get a ranked list from a survey:
- Rank order the criteria from 1 to 7, based on the importance they have on your buying decision.
- Select the three most important criteria for your buying decisions.
- Distribute 100 points among the seven criteria based on the importance each has on your buying decision.
Now let’s compare the three options so you can decide which one works best for your particular survey needs (we’re using a list of 7 criteria for these examples, but you could have any number of options).
Survey Question Options That Benefit Survey Respondents
#2 is clearly the easiest to answer for the respondent. The respondent merely has to divide the seven criteria into two groups, important or not important.
#3 is the next most friendly to the respondent because the respondent has the flexibility to decide the level of effort they want to put into the selections. If a respondent is so inclined, he or she can select his or her top three choices like alternative #2 and then just divide the 100 points among them. However, with this alternative the respondent has the flexibility to assign different point totals to the one or ones he thinks are important.
#1 requires the most effort from the respondent since each criterion must be given a separate rank. This can be difficult when two things seem equally important, or when the bottom of the list has three or four items that are equally unimportant but still must be ordered (i.e. given a rank).
Ease of Analysis and Interpretation for Options
Suppose we have 100 responses to the ranking question. For alternative #1 this yields 100 sets of 7 numbers (1 to 7). For #2 we have up to 300 “votes” and for #3 we have 100 divisions of 100 points into seven groups.
#2 and #3 are easily summarized and interpreted. One would just total the “votes” or “points” for each criterion to create a natural ranking of the seven criteria.
#1 is different. The summary is not so straightforward.
I could average the rankings for each criterion but I have to decide if I can live with a scheme where 100 4’s is the same as 50 1’s and 50 7’s. I could total the number of times each criterion got a first place rank, but then I have to be happy with a scheme that gives a higher ranking to a criterion with 10 first place ranks and no others in the top 3, than to one that got 9 first place ranks and 91 second place ranks.
Analysis and interpretation for option #1 takes some thought and rarely has an easy answer.
Which Ranking Option Should You Choose for Your Survey Question?
I like the simplicity and ease of analysis and interpretation of the select-your-top-one-third method (#2). It provides a natural ranking and is unlikely to overwhelm or frustrate the respondent. It is important to keep the respondent in a good frame of mind for answering the rest of the survey.
Clearly the distribute-100-points method (#3) provides more precision.
If I feel like I need more precision in distinguishing between items of a list I might use this method, but in most cases it is not really needed. Remember, it is always a good idea to consider how you are going to use the information you get back when deciding on a question to ask. In this case, one might want to consider what kind of precision is warranted.
Maybe we should rank order the alternatives!