Interpreting Trends in Your Survey Data

In my last article I showed ways to plot trend data or measurements over time (see Likert Scales and One Number Reporting).

I want to continue that discussion by talking about how to look at and interpret these survey data.

The Most Important Question to Ask About Survey Data

There is really only one question to ask when interpreting trend data and it is, “Has a change occurred?”

After that question is answered, there are a host of follow-on questions to ask, such as, “Why haven’t we improved?” or “Why is customer service satisfaction declining?” and so on. None of these questions can be addressed until we have determined whether or not a change has occurred.

Anytime we measure something more than once, we will see variation. Variation always exists. It is neither bad nor good.

That is why rather than asking the question, “Is there a variation?” the more pertinent question is, “Has a change occurred?” Even better, we can phrase the question slightly differently to read, “Is the variation we see in the data just normal variation or has something significant changed?”

Practice Looking at Trends in Your Survey Data

We don’t expect our measurements to always be the same even when nothing has changed, but how much do we expect them to vary?

There are statistical tools that can help to answer this question, but ultimately it takes practice looking at data; looking for and recognizing patterns, asking questions, investigating the numbers behind the data, and understanding how the data were collected. There is no substitute for experience when it comes to looking at trend data, or any data for that matter.

Let’s start by looking at the three examples in Figures 1, 2, and 3 below. Tips:

1. Ask “Am I looking at normal variation or is something significant going on?”
2. Try not to focus on individual points. Instead, look at the graph as a complete picture.
3. Look for patterns or anomalies.

There doesn’t seem to be a lot going on with this survey data. After three increases in a row starting in 2002 we might have been excited, but would have wanted to withhold the celebrating. The satisfaction level over the ten years looks fairly flat, hovering around 80%.

This example appears straightforward. Something changed for the better after 2007! The data from years 2001-2007 look relatively flat hovering around 58%, but the satisfaction levels after 2007 clearly increased.

This survey data looks erratic, especially when compared to the previous examples. Prior to the third quarter the response rate may have been higher and dropped, but we only have two data points to consider so not much can be said about that.

Next, the variability across quarters 3 through 8 seems large next to that for quarters 9 through 14. That should be investigated. The level of the response rate does not seem to have changed.

Optional: The Has-A-Change-Occurred (HACO) Chart

For those who want a more rigorous way to answer the question about whether a change has occurred, one can look at a HACO chart, perhaps better known as a control chart. The theory and application of HACO charts is quite extensive and complicated, but I’ve found one type of chart that is both easy to use and versatile. That chart is an Individuals Moving Range (IMR) chart.

An IMR chart, like all HACO charts, has an average line, an upper limit, and a lower limit. Calculating these values is straightforward, but it is never a good idea to make blind calculations without some understanding of the tool you are using. That is a discussion for another time.

To create an IMR chart, follow these steps:

1. Sum the absolute differences between one data point and the previous point (moving ranges of size 2).
2. Divide the sum obtained above by the number of data points minus one to create the average moving range, MR_ave.
3. Calculate the average of the points, X_ave.
4. Compute the lower limit as X_ave – 2.66 * MR_ave and the upper limit as X_ave + 2.66 * MR_ave.
5. Plot the data and add the average, upper, and lower limit lines to the chart.

Let’s see how the IMR chart looks for the three trend data examples given above.

The HACO chart tells us that nothing has changed with the data in Figure 4, that a change did occur in the data in Figure 5 and that the response rate has not changed in Figure 6. What about the issue of reduced variability that we saw in Figure 3? This is an important issue, but needs to be resolved by a different type of control chart, the Moving Range (MR) chart (which is a subject for another article).

In summary, let me remind you that it takes practice to read and interpret trend data. If you are not used to looking at graphical displays of data, it can be hard to interpret what you are seeing. Next time I’ll continue to look at the power of graphical displays in interpreting data.

PLEASE NOTE: The images and charts above are not part of Alchemer’s standard reporting tools. In order to create charts like this, you’ll need to export your survey data using our Export Survey Data to CSV feature, then use a spreadsheet such as Excel to analyze your data.

Additionally, for more complex data analysis, you can use our Export Survey Data to SPSS feature, then use SPSS to perform your analysis.