Now that you’ve carefully crafted your survey questions, it’s time to decide where to collect your answers from.
This is not a decision to be taken lightly. The way that you distribute your survey can introduce different forms of bias, and it will also influence the credibility of your data. Consider your target audience and determine the best way to reach them. Will you:
- Conduct a phone interview?
- Distribute your survey via social media?
- Send an email invitation to an existing email list?
- Seek targeted responses from a panel company?
- Intercept customers in person?
Choosing the Right Survey Distribution Method
To collect quality data, you need to survey your target audience. That sounds simple, but surveying the wrong audience (population) will give you the wrong answers.
Many companies run surveys in their customer base, which gives them a clear and finite audience. If you are not targeting your current customers, then you’ll be surveying a sample of the population.
Collecting Survey Data: Definitions
Before we move into some best practices for collecting great survey data, let’s get some definitions out of the way.
Sample Survey: a study that obtains data from a subset of a population in order to estimate population attributes in order to reduce the cost and/or work required to survey an entire target population.
Population: Represents the entire group of individuals for which you are trying to draw conclusions.
Sample: A sub-group of the population, chosen using statistically valid means, in order to represent the population as a whole.
Segmented Sample: A sample of a population that is broken down into segments that share characteristics.
Response Rate: Typically expressed as a percentage, this is the number of people who answer the survey divided by the number of people in the sample.
Survey Samples
When conducting a survey, you have two options: survey everyone, or survey just a percentage. As a general rule, you should always choose a sample of the population to survey. Here’s why:
- Surveying everybody is VERY expensive.
- You would contribute to the growing problem of cultural survey fatigue.
- No matter how hard you work, you will miss certain segments of the population.
- Using a statistically valid sample is just as effective.
What’s the Right Sample Size?
The number of responses you need for statistical accuracy depends on:
- How accurate you want your data to be (the margin of error or confidence interval).
- How repeatable you need the results to be.
- How large your total population is.
OR…400.
Surveying more than 400 people, regardless of your total population size, doesn’t increase your statistical accuracy. Generally you never need to survey more than 400 people for a single survey.
Sample Calculators
For those looking at a sample size smaller than 400, you can use a sample size calculator. These tools will use your total population, along with your desired confidence level and confidence interval, to suggest a statistically valid sample size.
The confidence interval, also called margin of error, is used to indicate the reliability of an estimate.
If you use a confidence interval of 5 and 60% of your sample picks an answer, you can be confident that if you asked that question to the entire population between 55% and 65% would pick the same answer.
The confidence level represents how often the percentage of the population who would pick an answer lies within the confidence interval. In the above example, a 95% confidence level means you can be 95% certain that 55-65% of respondents would pick an answer.
More on Sample Survey Audiences
If you’re segmenting your sample data for comparison, you need to make sure that the segments that you are using for comparison are the same as the segments in the represented population.
For example, when comparing men and women in the United States, you would need to make sure that the ration within your survey data was the same as the ratio within the larger US population.
Finding a Survey Audience
Option 1: Your Own List: You can pull respondents from your existing email list of customers, but DON’T use your entire customer base.
If you distribute surveys with any regularity you will very quickly reach survey fatigue in your entire population, and you may alienate your customers. Remember, if everyone has the same chance of being randomly selected, you are not biasing your results in any way.
Option 2: Panel Companies: These organizations sell anonymous survey responses to marketers and market researchers (and anybody else who needs them) to online respondents who match your target audience.
Panel companies will always do their best to target your specific audience, however, it’s difficult for them to account for every demographic out there. They also cannot account for changes in respondent circumstances that have not yet been communicated to the panel company.
It’s also important to keep in mind that some panel companies can produce respondent demographics after each respondent answered your survey for an additional fee, but this is not the standard practice for panel companies.
If you do not take proper precautions with your survey, you may risk never knowing this information.
To make sure that you collect demographic information, you’ll want to be sure to add demographic questions to your survey.
The last thing you’ll need to be aware of with using a panel company is to make sure that you place quotas in your survey for such demographic criteria as male/female or geographic ratios. Your demographic questions will need to be asked as survey questions so that your quotas are updated in real time as each panelist takes your survey.
You’ll also want to add disqualifying questions in your survey so that you can fully target the your unique audience and get the most accurate data.
Option 3: Incentives: You can offer incentives to gather responses from the larger population yourself, but this tactic can introduce bias into your survey.
People may not take the survey seriously, or the type of incentive you offer may attract certain types of respondents more than others. To protect your data, use these safeguards:
- Use a page timer with disqualification (responses don’t count if they are done too quickly)
- Create shorter surveys
- Add red herring questions (i.e. Answer “C” to this question. Disqualify anyone who doesn’t follow the instructions.)
- Clean your data! Good survey software can do this for you, or you can review it yourself for suspicious patterns like straight line answers.
Tips for Distributing Surveys to Your Customers
Don’t survey your entire customer list. Reduce survey fatigue by soliciting a small portion of your list for each survey that you run. Depending on your goal you may want to use a random sample or segment it.
You can create customer segments in lots of ways:
- New vs. long time customer
- By plan level or purchase type
- Random sample
- How recently they’ve completed a purchase
Now Go Forth and Collect Survey Data
You’re now fully armed with the information you need to make an educated on decision on your survey collection method.
It’s definitely not a decision to be taken lightly, and you’ll want to take every precaution possible to make sure that your data is protected and as accurate as possible.
This includes making sure that you use disqualifying questions and demographic quotas in your survey where necessary.
Taking these precautions will ensure that you will be making decisions off of quality data.
Running a survey can be hard.
Great Survey Design makes it easier.
Over many years and thousands of surveys, we here at Alchemer have made some discoveries. We’ve compiled our knowledge into this comprehensive Guide to Great Survey Design, which is yours with a simple click of a button: