survey for dissertation research

Work With Us

Private Coaching

Done-For-You

Short Courses

Client Reviews

Free Resources

Dissertation & Thesis Survey Design 101

5 Common Mistakes To Avoid (+ Examples)

By: David Phair (PhD) & Kerryn Warren (PhD) | April 2022

Surveys are a powerful way to collect data for your dissertation, thesis or research project. Done right, a good survey allows you to collect large swathes of useful data with (relatively) little effort. However, if not designed well, you can run into serious issues.

Over the years, we’ve encountered numerous common mistakes students make when it comes to survey design. In this post, we’ll unpack five of these costly mistakes.

Overview: 5 Survey Design Mistakes

Having poor overall survey structure and flow
Using poorly constructed questions and/or statements
Implementing inappropriate response types
Using unreliable and/or invalid scales and measures
Designing without consideration for analysis techniques

Mistake #1: Having poor structure and flow

One of the most common issues we see is poor overall survey structure and flow . If a survey is designed badly, it will discourage participants from completing it. As a result, few participants will take the time to respond to the survey, which can lead to a small sample size and poor or even unusable results . Let’s look at a few best practices to ensure good overall structure and flow.

1. Make sure your survey is aligned with your study’s “golden thread”.

The first step might seem obvious, but it’s important to develop survey questions that are tightly aligned with your research question(s), aims and objectives – in other words, “your golden thread”. Your survey serves to generate the data that will answer these key ideas in your thesis; if it doesn’t do that, you’ve got a serious problem. To put it simply, it’s critically important to design your survey questions with the golden thread of your study front of mind at all times.

2. Order your questions in an intuitive, logical way.

The types of questions you ask and when you ask them are vital aspects when designing an effective survey. To avoid losing respondents, you need to order your questions clearly and logically.

In general, it’s a good idea to ask exclusion questions upfront . For example, if your research is focused on an aspect of women’s lives, your first question should be one to determine the gender of the respondent (and filter out unsuitable respondents). Once that’s out of the way, the exclusion questions can be followed by questions related to the key constructs or ideas and/or the dependent and independent variables in your study.

Lastly, the demographics-related questions are usually positioned at the end of the survey. These are questions related to the characteristics of your respondents (e.g., age, race, occupation). It’s a good idea to position these questions at the end of your survey because respondents can get caught up in these identity-related questions as they move through the rest of your survey. Placing them at the end of your survey helps ensure that the questions related to the core constructs of your study will have the respondents’ full attention.

It’s always a good idea to ask exclusion questions upfront, so that unsuitable respondents are filtered our as early as possible.

3. Design for user experience and ease of use.

This might seem obvious, but it’s essential to carefully consider your respondents’ “journey” when designing your survey. In other words, you need to keep user experience and engagement front of mind when designing your survey.

One way of creating a good user experience is to have a clear introduction or cover page upfront. On this intro page, it’s good to communicate the estimated time required to complete the survey (generally, 15 to 20 minutes is reasonable). Also, make u se of headings and short explainers to help respondents understand the context of each question or section in your survey. It’s also helpful if you provide a progress indicator to indicate how far they are in completing the survey.

Naturally, readability is important to a successful survey. So, keep the survey content as concise as possible, as people tend to drop out of long surveys. A general rule of thumb is to make use of plain, easy-to-understand language . Related to this, always carefully edit and proofread your survey before launching it. Typos, grammar and formatting issues will heavily detract from the credibility of your work and will likely increase respondent dropout.

In cases where you have no choice but to use a technical term or industry jargon, be sure to explain the meaning (define the term) first. You don’t want respondents to be distracted or confused by the technical aspects of your survey. In addition to this, create a logical flow by grouping related topics together and moving from general to more specific questions.

You should also think about what devices respondents will use to access your survey. Because many people use their phones to complete your survey, making it mobile-friendly means more people will be able to respond, which is hugely beneficial. By hosting your survey on a trusted provider (e.g., SurveyMonkey or Qualtrix), the mobile aspect should be taken care of, but always test your survey on a few devices. Aside from making the data collection easier, using a well-established survey platform will also make processing your survey data easier.

4. Prioritise ethics and data privacy.

The last (and very important) point to consider when designing your survey is the ethical requirements. Your survey design must adhere to all ethics policies and data protection laws of your country. If you (or your respondents) are in Europe for instance, you’ll need to comply with GDPR. It’s also essential to highlight to your respondents that all data collected will be handled and stored securely , to minimise any concerns about the confidentiality and safety of their data.

Since many respondents will be completing your survey on their phones, it's very important to ensure that your survey's mobile-friendly.

Mistake #2: Using poorly constructed questions

Another common survey design issue we encounter is poorly constructed questions and statements. There are a few ways in which questions can be poorly constructed. These usually fall into four broad categories:

Loaded questions
Leading questions
Double-barreled questions
Vague questions

Let’s look at each of these.

A loaded question assumes something about the respondent without having any data to support that assumption. For example, if the question asks, “Where is your favourite place to eat steak?”, it assumes that the respondent eats steak. Clearly, this is problematic for respondents that are vegetarians or vegans, or people that simply don’t like steak.

A leading question pushes the respondent to answer in a certain way. For example, a question such as, “How would you rate the excellent service at our restaurant?” is trying to influence the way that the respondent thinks about the service at the restaurant. This can be annoying to the respondent (at best) or lead them to respond in a way they wouldn’t have, had the question been more objective.

Need a helping hand?

A double-barreled question is a question that contains two (or more) variables within it. It essentially tries to ask two questions at the same time. An example of this is

“ Do you enjoy eating peanut butter and cheese on bread?”

As you can see, this question makes it unclear whether you are being asked about whether they like eating the two together on bread, or whether they like eating one at a time. This is problematic, as there are multiple ways to interpret this question, which means that the resultant data will be unusable.

A vague question , as the name suggests, is one where it is unclear what is being asked or one that is very open-ended . Of course, sometimes you do indeed want open questions, as they can provide richer information from respondents. However, if you ask a vague question, you’ll likely get a vague answer. So, you need to be careful. Consider the following fairly vague question:

“What was your experience at this restaurant?”.

A respondent could answer this question by just saying “good” or “bad” – or nothing at all. This isn’t particularly helpful. Alternatively, someone might respond extensively about something unrelated to the question. If you want to ask open-ended questions, interviewing may be a better (or additional) data method to consider, so give some thought to what you’re trying to achieve. Only use open-ended questions in a survey if they’re central to your research aims .

To make sure that your questions don’t fall into one of these problematic categories, it’s important to keep your golden thread (i.e., your research aims, objectives and research questions ) in mind and consider the type of data you want to generate. Also, it’s always a good idea to make use of a pilot study to test your survey questions and responses to see whether any questions are problematic and whether the data generated is useful.

If you want to ask open-ended questions, you may want to consider complementing your survey with a small round of interviews.

Mistake #3: Using inappropriate response types

When designing your survey, it’s essential to choose the best-suited response type/format for each question. In other words, you need to consider how the respondents will input their responses into your survey. Broadly speaking, there are three response types .

The first response type is categorical.

These are questions where the respondent will choose one of the pre-determined options that you provide, for example: yes/no, gender, ethnicity, etc.

For categorical responses, there will be a limited number of choices and respondents will only be able to pick one. This is useful for basic demographic data where all potential responses can be easily grouped into categories.

The second response type is scales .

Scales offer respondents the opportunity to express their opinion on a spectrum . For example, you could design a 3-point scale with the options of agree, neutral and disagree. Scales are useful when you’re trying to assess the extent to which respondents agree with specific statements or claims. This data can then be statistically analysed in powerful ways.

Scales can, however, be problematic if they have too many or too few points . For example, if you only have “strongly agree”, “neutral” and “strongly disagree”, your respondent might resort to selecting “neutral” because they don’t feel strongly about the subject. Conversely, if there are too many points on the scale, your respondents might take too much time to complete the survey and become frustrated in the process of agonising over what exactly they feel.

The third response type is the free form text box (open-ended response).

We mentioned open-ended questions earlier and looked at some of the ways in which they can be problematic. But, because free-form responses are useful for understanding nuances and finer details, this response type does have its benefits. For example, some respondents might have a problem with how the other questions in your survey are presented or asked, and therefore an open-ended response option gives them an opportunity to respond in a way that reflects their true feelings.

As you can see, it’s important to carefully consider which response types you use, as each one has its own purpose, pros and cons . Make sure that each response option is appropriate for the type of question and generates data that you will be able to analyse in a meaningful way.

It’s also good to keep in mind that you as the researcher will need to process all the data generated by the survey. Therefore, you need to consider how you will analyse the data from each response type. Use the response type that makes sense for the specific question and keep the analysis aspect in mind when choosing your response types.

It's essential to use the best-suited response type for each question to ensure the data that you collect is both meaningful and analysable.

Mistake #4: Using poorly design scales/measures

We’ve spoken about the design of the survey as a whole, but it’s also important to think carefully about the design of individual measures/scales. Theoretical constructs are typically measured using Likert scales. To measure these constructs effectively, you’ll need to ensure that your scales produce valid and reliable data.

Validity refers to whether the scale measures what you’re trying to measure . This might sound like a no-brainer, but oftentimes people can interpret questions or statements in diverse ways. Therefore, it’s important to think of whether the interpretations of the responses to each measure are sound relative to the original construct you are measuring and the existing literature relating to it.

Reliability, on the other hand, is related to whether multiple scales measuring the same construct get the same response (on average, of course). In other words, if you have three scales measuring employee satisfaction, they should correlate, as they all measure the same construct. A good survey should make use of multiple scales to measure any given construct, and these should “move” together – in other words, be “reliable”.

If you’re designing a survey, you’ll need to demonstrate the validity and reliability of your measures. This can be done in several ways, using both statistical and non-statistical techniques. We won’t get into detail about those here, but it’s important to remember that validity and reliability are central to making sure that your survey is measuring what it is meant to measure.

Importantly, when thinking about the scales for your survey, you don’t need to reinvent the wheel. There are pre-developed and tested scales available for most areas of research, and it’s preferable to use a “tried and tested” scale, rather than developing one from scratch. If there isn’t already something that fits your research, you can often modify existing scales to suit your specific needs.

To measure your theoretical constructs effectively, you’ll need to ensure (and show) that your scales produce valid and reliable data.

Mistake #5: Not designing with analysis in mind

Naturally, you’ll want to use the data gathered from your survey as effectively as possible. Therefore, it’s always a good idea to start with the end (i.e., the analysis phase) in mind when designing your survey. The analysis methods that you’ll be able to use in your study will be dictated by the design of the survey, as it will produce certain types of data. Therefore, it’s essential that you design your survey in a way that will allow you to undertake the analyses you need to achieve your research aims.

Importantly, you should have a clear idea of what statistical methods you plan to use before you start designing your survey. Be clear about which specific descriptive and inferential tests you plan to do (and why). Make sure that you understand the assumptions of all the statistical tests you’ll be using and the type of data (i.e., nominal, ordinal, interval, or ratio ) that each test requires. Only once you have that level of clarity can you get started designing your survey.

Finally, and as we’ve emphasized before, it’s essential that you keep your study’s golden thread front of mind during the design process. If your analysis methods don’t aid you in answering your research questions, they’ll be largely useless. So, keep the big picture and the end goal front of mind from the outset.

Recap: Survey Design Mistakes

In this post we’ve discussed some important aspects of survey design and five common mistakes to avoid while designing your own survey. To recap, these include:

If you have any questions about these survey design mistakes, drop a comment below. Alternatively, if you’re interested in getting 1-on-1 help with your research , check out our dissertation coaching service or book a free initial consultation with a friendly coach.