Common Survey Mistakes

Common survey mistakes to avoid

There are many common survey mistakes if you design surveys and manage survey projects. Let’s have a look at some of them.

Fail to create the right context for the survey respondents

To engage survey respondents, make them want to complete the survey and ensure they give valid responses, you must create the right context. This is essential. How you introduce the survey and how you go about collecting the feedback can have a major impact on the survey results.

Fail to establish and communicate clear survey policies

Clearly defined and communicated survey polices are essential in a survey project. Policies guide the actions and decisions of those who manage the survey project and protect the respondents. Policies should consider things like anonymity, how the data will be used, who will have access to the data, etc.

Fail to pilot the survey

Always conduct a survey pilot. A pilot is an opportunity to get feedback on the communications, instructions, survey questions and the scales used along with the survey structure and usability on the devices likely to be used. Based on the pilot feedback, you can then refine the survey and the communications and address any issues identified.

Send the survey to a large group without starting with a small group

Wherever possible, don’t send the survey to a large number of respondents without first sending it to a smaller group. Confirm that everything is fine with the smaller group before sending to the large group.

Ask irrelevant or unnecessary questions

Don’t add questions that you don’t need to ask. Don’t throw in a bit extra to the survey, just because you can. Stick with questions that support your survey goal. This is one of the most common survey mistakes.

Introduce bias

If there is an answer that you’re hoping to get, you can bias your questions so that you get it. Leading questions is the easiest way to do this. An example would be, “How happy are you with our fantastic service?” In that question, you tell the respondent that the service is fantastic. A better approach would be to ask about the service and then use a rating scale that allows the respondent to assess the service level.

Create survey fatigue and tire out your respondents

Have you come across the term ‘survey fatigue’? Ways to do this include:

  • Ask too many questions
  • Put a lot of questions on every page
  • Ask long questions
  • Ask confusing questions
  • Ask overly complex questions

Piloting the survey will give you useful information as to whether the survey is tiring.

Poor Communication

When respondents don’t understand survey questions, it will make the survey data worthless. Don’t use complicated language that is ambiguous or full of jargon. Poor communication is another one of the very common survey mistakes.

Assume respondents know more than they do

Don’t leave room for ambiguity and don’t rely on prior knowledge. Also, don’t ask respondents to remember things from some time ago.

Ask two questions in one

Make sure your survey questions contain mutually exclusive ideas.

Ask too many open-ended questions

Choose open ended questions wisely and don’t over-use them. Open-ended questions are time consuming. They also involve much work analyzing the responses. So, be prudent in asking open-ended questions

Lack of attention to detail

Pay attention to the detail in the survey. Poor grammar and spelling errors are unprofessional. Details like this send a message about the importance of the survey. Minute details in grammar can also affect the validity of the data.

Inadequate response options

Make sure that the response options cover the needs of the target group of respondents. It can be very frustrating to be forced to select and not have a relevant choice.

Use too many different rating scales

Using many different rating scales has two draw backs. First, it becomes difficult and time consuming for respondents to answer the survey. Second, interpreting the results becomes difficult. A standard rating scale allows easier comparison of responses across questions.

Use too many response options

In multiple response questions, don’t have too many response options. Choosing answers from long lists is difficult and can lead to respondents not taking the time to respond accurately. So, keep the response options reasonable.

Use the wrong page structure for the type of survey and the audience

Carefully consider the length of the survey, the type of audience and the device that the respondents are likely to be using when they complete the survey. Then design the structure to suit. For example, long surveys can be very frustrating if only one question is placed on each page. On the other hand, placing multiple open ended questions on one page creates the risk of respondents losing what they have entered if they are interrupted and they have not saved their responses.

Force responses to every question

Do not overuse the mandatory questions. Use them sparingly. You may be forcing people to make a selection that they don’t want to make and thereby invalidating the data. Forcing responses can be very frustrating. And, if someone does not really want to answer a question then they may just choose any response so they can move on.

Not labelling rating scale steps

We prefer not to use long rating scales like a 10-point scale. These scales are another one of the common survey mistakes we see. Where possible, each point in a rating scale should be labelled to give meaning (albeit subjective) to the scale points. A 10-point scale with Highly Unlikely at one end and Highly Likely at the other end can be frustrating. And, when there are no labels for the options 2 to 9 in between then we question the point. Five point scales and seven point scales are preferable. Not labelling rating scales is one of the common mistakes apparent in many surveys.

Fail to provide a ‘Not Applicable’ or ‘Don’t Know’ option

Consider the audience and the questions being asked. In many cases, you should include an option for the respondent if they do not have experience with the question being asked. It is often unreasonable to expect that respondents have relevant experience to respond to every question in the survey. So, consider including a Don’t Know or Not Applicable option in the rating scale.

Use this resource as a checklist to avoid the common survey mistakes.