Misinformed or poor data can have wide-reaching negative ripple effects on a product and company. Product Managers know that the data and research they provide informs critical product decisions.
A key component to delivering reliable data is by taking all necessary steps to avoid survey bias. Survey bias can be mitigated in several ways during the design and implementation phase for product surveys.
In this article, we review the different types of bias and how to avoid damaging data results.
Sampling Bias
When some members of your survey population are less likely to be invited to be surveyed than others.
Who are you trying to reach with your survey? If it’s all of your customers, then you will need to conduct your survey in a way that all customers have a chance to respond. For instance, sharing a link through social media may limit the number of customers that see your survey.
The easiest way to reduce sampling bias is to share your survey through several different communication channels. Send your survey out through an email campaign, share it on your website, post QR codes in your physical locations.
To reach everyone you may even need to consider hiring telephone or in-person interviewers to recruit respondents who don’t have access to the internet or who don’t spend much time on the web. You can also mail out paper surveys to segments of your population who are not particularly tech-savvy.
Nonresponse Bias
When respondents differ in meaningful ways from nonrespondents.
Regardless of how careful you are about inviting everyone to respond, there may be some people who are either unwilling or unable to respond. These people are often systematically different than those who do respond, and you do not want to miss out on this segment as it will throw your results off or invalidate them entirely.
If you have are asking your customers for sensitive information there may be a very particular segment of your target survey respondents that may feel uncomfortable filling out the survey and will be more likely to refuse, creating bias towards a particular type of customer only.
Another way to create nonresponse bias is because the invite to engage in the survey was not properly tested. For instance, if a survey is sent out via email, and greater numbers of people today check email via their phones, a segment of your customers may not complete the survey if it is not mobile-friendly both in length and format.
Here are a few ways to mitigate nonresponse bias:
- Thoroughly pretest your survey mediums to ensure that surveys are formatted and compatible for all devices.
- Allow ample time for data collection. Do not rush the data collection period. It is recommended to extend a survey collection period to at least two weeks so that participants can choose any day of the week to respond according to their busy schedule.
- Ensure confidentiality if the survey involves sensitive questions
- Use incentives, especially for surveys that will take more than 5 minutes to complete.
- Send notifications and reminders about the survey; this can include a pre-notification email with information about the survey to come, a personalized invite, and other reminders.
Response Bias
When subconscious and conscious cognitive factors result in less-than-truthful responses.
Even once you someone is taking the survey there are several ways they may respond in a biased way.
Acquiescence Bias
Yea-saying is a bias that found in agree/disagree type questions.
In this case, people are more likely to be agreeable in a survey more than they typically would be.
Demand Characteristics
Bias can also take the form called Demand Characteristics, which means that simply by being a part of the survey a respondent may change their typical response to align with the purpose of the survey.
Extreme Responding
Being more likely to select extreme options occurs when a respondent is presented with a scale, for instance, a 5-point scale from 1 to 5. Interestingly, the opposite of this bias occurs when participants only select neutral responses.
This tendency towards neutral or extreme responses is culturally specific.
Desirability Bias
This kind of bias occurs when respondents are motivated to answer in such a way that they ascribe to behaviors and characteristics that are desirable and deny undesirable characteristics and traits.
Reduce response Bias by conducting your surveys online. In this way it automatically reduces the potential for response bias because the questions are self-administered thus making it easier, to be honest.
Here are some useful tips for reducing response bias:
- Ask neutrally worded questions
- Make sure your answer options are not leading
- Make your survey anonymous
- Remove your brand, as this can tip off your respondents on how you wish for them to answer
Question Order Bias
Influencing respondent answers due to the order in which questions are asked.
The order of both questions and answer options in your survey matters! Questions that come early in your survey may influence how respondents respond to questions later in your survey.
For example, asking overall satisfaction after specific satisfaction questions may bias the responses. The response is being primed to answer in a certain way by leading up to the overall satisfaction question.
In addition to the bias that is introduced by order of questions, the order of answer options can also bias your results. In self-administered surveys respondents usually prefer the first few response choices in a list. For phone surveys and in-person interviews where an interviewer is reading the answer options aloud, respondents tend to choose the latter options.
Reduce Order Bias in the following ways:
- Keep the number of scale questions to a minimum
- Group your survey by topic
- Leave demographic questions until later in the survey
- Ask question that engage respondents
- Randomize your question and answer options
Bias Comes in Many Forms. Remove Them All.
Bias is tricky. Even professionals who have been leveraging surveys for their entire careers will find themselves getting trapped by survey bias.
By remaining vigilant and carefully designing and implementing your surveys, you can create better, more accurate, more helpful surveys in the future.
Happy survey creating!