Many people jump directly into the “build” phase of their survey, but in reality once you’re writing questions you’re nearly halfway done with your project.
Survey questions are like the walls, floors, doors, and windows of a house: they’re vital, but if you put them up without laying a foundation you’re in for some serious trouble.
So, if you haven’t already, check out these previous articles on identifying your survey goals and considering your survey design before you start building.
Ok, ready to move forward? Great. Time to create a great survey.
Constructing Survey Questions
Survey design involves thinking about the psychology, emotions and words behind the questions. The design process is the strategic phase. It requires the Big Picture, including your survey goals and learning objectives.
Actually building the survey, on the other hand, is the tactical phase. It takes into account the logistical issues like security, logic, survey fatigue, bias, and data collection.
During survey builds you think about the nitty gritty, including question types, survey length, anonymity, and data analysis.
Turning Goals Into Real Survey Questions
Generally a small team of 1-3 are involved in constructing a survey, though they should certainly take stakeholder input into account during the process.
Survey design — particularly the identification of goals and objectives — can be a group effort, but having too many workers on a construction site can create more problems than it solves.
Ideally your survey construction team will consist of the people who will be responsible for presenting the data, because they need to understand where it came from in order to talk about it intelligently. Other good team members may be those who are going to be acting on the data; they’ll often have different insights into question phrasing and order.
Qualitative vs. Quantitative Questions
One of the biggest questions about questions is whether to use qualitative or quantitative question types. The answer to this question depends on what you want to achieve.
Qualitative questions are open ended. They usually include a “why” somewhere, and they can be very useful in helping to define a problem.
Quantitative questions are designed to simply gather data, not ask opinions.
Both are highly useful, but you should choose between them carefully because they offer very different kinds of data.
Qualitative Questions
Qualitative questions help define a problem. They ask “WHY?”
If you are exploring a hypothesis, a qualitative survey can identify a problem and its nuances before conducting a quantitative survey. Qualitative question are open-ended, which means text analysis is required to interpret results (and these are particularly susceptible to interpretation bias).
Qualitative Question Types
Qualitative questions are always open text questions, but they come in many forms, including:
- Text Box: responses can be one word to one sentence long
- Essay Box: several sentences to several hundred words, depending on the limit you set (or don’t set)
When asking a qualitative question, consider using an autocomplete feature to minimize data cleanup. This lets you suggest common answers so that you get consistent responses, but it can also introduce bias, so proceed with caution.
It’s worth trying, however, because having the same answer format will make it much easier to analyze the data.
For instance, these answer options all mean the same thing to a human respondent:
- twenty-five
- 25
- twentyfive
- Twenty-Five
But each format (including upper case versus lower case) will be treated as a different answer option during open text analysis.
Reporting on essay questions can be especially challenging. Some tools will help you by creating word clouds of common terms or performing open text analysis, but to get the full impact long-form questions really need to be read individually.
Quantitative Questions
Quantitative question ask “WHAT”, “WHEN”, or “HOW”.
These questions typically quantify a pre-defined problem so you can understand how prevalent it is. Quantitative questions have limited answer options, which makes it easier to measure the results.
Quantitative Question Types
These are the most commonly used quantitative survey questions, and your respondents will know how to deal with them. There are less common, more advanced varieties of quantitative questions, but keeping it relatively simple will create a better experience for those answering your survey.
Common question types include:
- Radio Button: Use these when you want a single answer option.
- Check Box: Use these when multiple answer options are acceptable.
- Drop Down: These are most commonly used as single select, but can also be used as multi-select answer options. Use these to save space when presenting a long list of answer options.
- Likert Scale: These are most commonly used for measuring emotions such as satisfaction or agreement. Odd numbered scales allow for a neutral response.
Avoiding Survey Fatigue
Online surveys shouldn’t be exhausting to take. Tired respondents will either abandon your survey or give you substandard data. This means that when you’re building your survey, you need to take the respondent’s experience into consideration.
Having a long list of items to rank generally increases fatigue and dramatically increases drop-offs.
Many surveys fail to collect useful data simply because they were not designed to keep their respondents interested. As a survey builder, it is your job to reduce survey fatigue whenever possible while still gathering solid data that your team can act on.
The first step is to choose your question types carefully! Limit the number of qualitative questions, which are far more fatiguing to answer than their quantitative cousins.
General Question Guidelines
You also need to make sure that people taking your survey don’t get thrown by awkward answer options or question construction.
A common error is creating overlapping answer options. Selecting one choice should completely exclude all the others; when answers intersect it can cause a lot of confusion.
For example:
Correct format:
How long have you been a member?
1-10 years
11-20 years
21–30 years
31+ years
Incorrect format:
How long have you been a member?
1-10 years
10-20 years
20–30 years
30+ years
You’ll also want to refrain from using double-barreled questions, which combine multiple questions into one. This adds confusion and skews your data.
For example:
Don’t ask:
How satisfied are you with our buffet food and drink options?
Instead, ask:
How satisfied are you with our buffet food options? Address drink in a separate question.
Finally, ensure that you provide inclusive answer options for required questions. If you’re not sure that you’ve included all the possible responses in your answer choices, include an “other” option for those who don’t find the right choice in your list. A forced answer that doesn’t apply will taint your results.
The Importance of Survey Validation
Validation is the process of checking your survey to ensure it meets your specifications and fulfills its intended purpose.
Like editing a document, validation requires a detailed review of answer options, logic, reporting values, and reporting data to verify that you are setup to collect quality data.
It can be time-consuming, but the benefits you’ll see in your data quality make it more than worth the effort!
How to Validate Your Survey
Validation is a key component of great survey design, but it’s often overlooked as people skip straight to testing. Testing can uncover some problems with a survey, but validation is a more rigorous review process.
A common problem occurs when different data formats are treated as different answer options. This will make it hard to analyze the data unless we do some data cleaning to standardize the answer format.
Discovering these issues earlier rather than later will save a lot of time and headache down the road, and it’s not an issue that is likely to come up during a standard test.
Changing the question type, adding instructional text on the proper format you would like entered, using autocomplete, or using a data validation feature will go along way in solving this common problem.
Testing Your Survey
Testing your survey simply involves taking your survey on the different devices that your respondents will be using to ensure it displays and flows correctly.
Advanced survey tools provide a testing feature that quickly generate test data so that you can also look at the results to see if it reports as you expected.
Still, you should have stakeholders, colleagues, and friends take your survey for thorough testing.
Run a few reports on the data, then ask yourself these burning survey questions:
- Are your questions reporting the way you expect?
- Are you able to create the reports you need using the data you’re collecting?
- Is the data in the format you need?
The Power of Survey Logic
One of the coolest parts of building a survey is adding logic. It’s like including little magic tricks that change how the survey acts for different respondents.
Put simply, logic is a set of conditions that you can apply to a question, answer, or even an entire page of your survey that affects how it performs.
For example, you might ask if your respondents had an appetizer last time they were at your restaurant. If they answer “No,” you could use logic to skip them past all the questions about appetizers.
Survey logic is extremely powerful, and its benefits come in two flavors:
Fatigue Fighting: Keep respondents engaged by only showing them questions that are relevant to them.
- Page-jumping: Skip entire pages that aren’t relevant to a respondent.
- Show-when logic: Only show questions when particular conditions are met.
- Percent branching: Randomly assign a set percentage of your respondents to a branch of your survey.
- Piping (a.k.a. repeating): Inserting data collected early in the survey into a later question.
Bias Fighting: Avoid any bias that might come from your question or content order.
- Randomization: Randomize question and/or answer options.
- Disqualification: Prevent those who don’t qualify from answering your survey to collect cleaner data.
- Survey timing: Identify and disqualify survey responses that were answered too quickly.
- Vote protection: Prevent respondents from taking your survey more than once.
Survey logic is one of the best ways to keep your survey relevant and collect quality data.
A Well-Built Survey is a Successful Survey
Approaching your survey build with care and attention will make sure that it serves your ultimate purpose. By creating an appealing design and an experience that’s as personalized as possible, you’ll get more engaged respondents who give you better data.
Keep these best practices in mind and you’ll be well on your way to building a beautiful survey on the great design foundation that you’ve created.