Which of the following occurs when respondents believe that past events happened more recently than they actually did?
QUESTIONNAIRE DESIGN Show Introduction A poorly designed questionnaire can be the biggest source of non-sampling error (either directly or indirectly). The questionnaire can influence the response rate achieved in the survey, the quality of responses obtained and consequently the conclusions drawn from the survey results. Questionnaire design should be started by considering the objectives of the survey and the required output, and then devising a list of questions to accurately obtain this information. Careful consideration should be given to a number of factors including the types of questions to be asked, the questionnaire wording, the structure and design of the questionnaire and testing the questionnaire to ensure that quality data is collected. This section discusses the developmental phase of questionnaire design, question wording, different types of questions and the structure and design of the questionnaire. Testing the draft questionnaire is covered in Survey Testing. Questionnaire Development This development phase needs to be undertaken to ensure that the data items in the final questionnaire are appropriate and can be collected accurately. It is also important to consider early on, the coding of questionnaire responses and the type of processing system to be used to process the questionnaire. Defining Collection Objectives with the Client outlined the importance of clearly establishing the research objectives for
the collection of any information. Similarly, in the process of questionnaire development, the reasons for collecting the information need to be clearly articulated. Defining the collection objectives with the client includes:
A similar process of defining and justifying the content in consultation with users should be conducted when adding new or redeveloped questions to an existing collection. Researching the Topic As stated in The Set-up Stage of a Survey, it is very important that the researcher who is going to design the questionnaire undertakes background research into the topic under consideration. In terms of questionnaire design, the research should aim to:
It is also worthwhile looking at past collections of the same topic to learn from past experience. Such collections can prove to be a useful basis for the current collection and also help to avoid possible mistakes. It is important to look at the concepts, definitions and question wording of past collections if a time series is to be created, since changes in these will result in changes in responses. For example, a question in the 1981 Population Census, 'Do you have a mortgage?' received very different responses from those that were obtained in the 1986 Census when the question was changed to 'Have you paid off your house?' Many people who had second mortgages included this in the 1981 Census but not in the 1986 Census. Past collections of similar topics may also be worth looking at. For example, if you are collecting information on drug use it may be appropriate to follow a similar methodology to that used in surveys of alcohol use. In examining past collections the method of collection used should also be considered. Some topics will be more or less appropriate for particular method. General Considerations The main functions of questionnaires are to extract information from respondents, record and process it. To achieve this effectively, a questionnaire should:
Additionally, it is necessary to realise that a questionnaire often cannot provide all the information that a client would like to obtain. If a questionnaire becomes too long or confusing, respondents may be unwilling to complete it, or they may make mistakes. TYPES OF QUESTIONS Questions can be classified into types based on their answer formats and also the type of data sought. Answer Formats Questions can generally be classified as one of two types - open or closed - depending on the amount of freedom allowed in answering the question. When choosing the type of questions, consideration should be given to factors such as the kind of information which is sought, ease of processing, and the availability of the resources of time, money, and personnel. Open Questions Closed Questions
Limited choice questions are those which require a respondent to choose one of two mutually exclusive answers. For example, yes/no. These are questions where the respondent is required to choose from a number of responses provided. Checklist questions allow a respondent to choose more than one of the responses provided. These questions provide a set of responses where the last alternative is 'Other, please specify'. Partially closed questions are useful when it is difficult or impractical to list all possible choices. Choosing Between Questions Types In choosing between these two alternatives (open or closed questions), consideration should be given to factors such as the data requirement, the kind of information sought, the level of accuracy needed, ease of processing and the availability of coding resources, the position of the questions on the form and the sensitivity of the question. In general, closed questions are better for both the interviewer and the coder. Once the questions have been chosen, they should be tested and retested until the best choice has been made. This is covered in more detail in Survey Testing. Type of Data Sought There are six main types of information or data that can be obtained from questions. They are discussed below. Factual Questions Opinion or Motivational Questions
Behavioural Questions Hypothetical Questions Classification or Demographic Questions Knowledge Questions QUESTION WORDING AND RESPONSE CATEGORIES There are a number of factors to consider when designing questions to ensure that appropriate answers are obtained. Several aspects of question design can introduce error, namely: Language Technical language or jargon should only be used in cases where it is part of the normal language of the survey's target population. An example of this case would be a survey of information technology specialists: the survey would need to use language that is 'jargon' to the survey designer, but appropriate for the respondent. A general principle to keep in mind is that the wording of questionnaire items should be specific, definitive, consistent, brief, simple and self-explanatory. Ambiguity For example, consider the question 'Has your standard of living decreased substantially because of a sharp increase in your monthly mortgage repayments?' A 'No' answer could mean any one of a number of things - for instance: 'No my standard of living has not dropped because of increased repayments' or 'No, my repayments have not increased'. A question may also seem straightforward, but allow for a variety of different kinds of answers. It is important to include the measurement unit you require wherever one applies, e.g. dollars, days, litres. Double-Barrelled
Questions (Multiple Concepts in one Question) Leading Questions Unbalanced Questions Recall/Memory Error Subjects which are of greater importance or interest to respondents, or events which happen infrequently, will be remembered over longer periods and more accurately. Where possible (eg. with financial information), questions should be framed so that respondents can refer to their own records which would enhance accurate reporting. Minimising the recall period also helps to reduce memory bias. A specific type of memory error is telescoping. This occurs if respondents report events as occurring either earlier or later than they actually occur, incorrectly bringing events into the reference period. This effect is alleviated somewhat by being very specific about when the reference period begins and ends, for example using "the week ending Saturday 1st September" rather than "last week". Intrusive (Sensitive) Questions Business survey respondents can also find some topics sensitive, such as IT security breaches or donations to charity, as well as not wanting to reveal commercial-in-confidence information about their business. Business surveys add a new dimension to collecting sensitive data as it is often necessary for different respondents, sometimes from different areas, to complete parts of the form and approve its content. Some respondents might not want the others to see particular answers. The negative effect of sensitive questions may be aggravated if they are placed at the beginning of the questionnaire and can therefore contribute to non-response if respondents are unwilling to continue with the remaining questions. If a sensitive question is further into a form the respondent is more committed to completion, and if they do refuse to continue, the partial response is more useful. Ways of overcoming difficulties associated with sensitive questions may include reassuring respondents that the information they provide is confidential, and not requiring respondents to write their name anywhere on the survey form. Attitude Strength Care needs to be taken with the use of attitudinal scales because some respondents may have difficulty interpreting the scale. Additionally, such scales are interpreted subjectively and this interpretation can differ between respondents. Acquiescence This tendency can be due to a combination of factors, such as the personality and education level of respondent, as well as conditions of the interview or design of a self-completed questionnaire. Respondents will often agree when the question is ambiguous or otherwise difficult to answer. The effect may be exaggerated when the respondent is fatigued or has to answer a long string of questions with the same response categories. A related effect is satisficing, where respondents select the first reasonable answer rather than make the effort to find or remember the best answer. Adequate Response Categories
This provides a problem for those respondents whose age is 20. Another problem that could arise is overlapping response categories. Ranges should always be mutually exclusive. For example:
This provides a problem for respondents whose age is 20 since they could respond in either or both categories. Response categories also need to be worded carefully, as respondents will use them to clarify or extend the meaning of the question. For example, if a question used a frequency scale with five points "Never", "Rarely", "Average", "Often", and "Frequently" a respondent may incorrectly assume the scale represents the population distribution. If they consider themselves to be normal or extreme compared to the population on the activity of interest their answers will differ regardless of the actual frequency they engage in the activity. Number of Response Options Don't Know Category Tone 'Do you think that gun ownership should be forbidden?' or 'Do you think that gun ownership should not be allowed?' Minor changes in wording can also have a significant affect on responses. One should therefore be careful when looking at alternative wordings. The use of negative words like "not" should be avoided in questions as they are easily missed by respondents. In addition, using "not" in a scale such as "Satisfied", "Neither" and "Not satisfied" doesn't provide a true opposite. "Dissatisfied" would be a better alternative, however "Unsatisfied" could also be used and would mean something slightly different to respondents. STRUCTURE AND LAYOUT OF THE QUESTIONNAIRE Not only does the wording of questions require attention to detail, but also the 'look' of the questionnaire. Poorly designed questionnaires (eg. hard to read text) not only serve as a disincentive to respondents completing the questionnaire, but can also result in respondents making errors. Some of the important elements of questionnaire structure and layout are outlined below. Sequencing The remaining questions should be logically structured so that the interviewer or respondent does not need to alternate between pages of the questionnaire. For example, any explanatory notes should be presented as part of the question they refer to, not on a separate page. Questions which may be sensitive to respondents should generally not be placed at the beginning of a questionnaire. Rather, they should be placed in a section of the form where they are most meaningful to the context of other questions. In this way the format of the questionnaire can act as a buffer to help the respondent feel more comfortable with sensitive questions after establishing rapport. In self-enumeration questionnaires, to ensure that respondents answer only those parts of the questionnaire that are relevant, filter questions may be used to direct respondents to skip the questions that do not apply to them. Filter questions are also used in interviewer based surveys to direct interviewers to follow a series of questions according to answers given by respondents. Filter questions need to be used with care as respondents (and interviewers) need to have sufficient information about the skip condition to judge whether the respondent should skip. Filters should also generally be avoided for sensitive topics as respondents will tend to give the answer that avoids answering the sensitive questions. If the instructions are not clear and straightforward, interviewers or respondents can follow an incorrect sequence or miss questions. In general, only one or two conditions should be placed in each sequence guide. Computer-assisted interviewing and electronic self-completion forms can make complex sequencing much easier. Filter questions also identify sub-populations. For example: Q7 'Were you born overseas?' Order of Questions Order of Response Options If possible, options should be presented in a meaningful order. If some options are more socially desirable than others these should go last to reduce bias. For example, an education question should present the qualifications in order from lowest to highest. For some self-completed lists, alphabetical order is the most appropriate to help the respondent find which option they want, for example if respondents have to select which crops they produce. Response Options and Respondent Difficulties Length Questionnaire Layout Particularly for those questionnaires which are completed by respondents, the design may contribute to errors as a result of:
• violating the normal reading path of the respondent (English readers expect to read from top left to bottom right and e.g. big headings and bright pictures in the middle of pages disrupt this); • instructions which can be easily overlooked (e.g. those which are not clearly differentiated from the questions or are not placed near the relevant part of the question); and • inadequate space for answers. A questionnaire should be built up question by question. That is, it is inappropriate to start with the page and then try and make the questions fit.
• should have a good appearance as it might affect the response, i.e. the questionnaire should be well designed and presented and therefore easy to answer; • should clearly identify the date, title and the organisation; • should clearly outline the purpose of the survey; • should assure respondents about the confidentiality of the information they are providing; • should provide a contact number so that respondents can obtain help if they require it and a due date; and • should have pages that are numbered consecutively, with a simple numbering system. Specific principles of good questionnaire design are outlined below:
• typeface: avoid ornate and decorative typefaces. Serif type fonts (eg. Times) are easier to read for questions (on paper) than sans serif types (eg. Helvetica); • upper case text is difficult to read, and should be avoided where possible; • allow enough space for answers; • tick boxes are a popular way of obtaining responses. Where possible, they should be vertically aligned rather than horizontally aligned; and • lines are sometimes useful in order to divide columns, sections and questions. These lines should be as fine as possible and they should only be used where necessary; • keep the amount of ink on the form to the minimum necessary for the form to work properly. Cluttered forms contribute to respondent fatigue and errors, thus leading to a decline in data quality; • black text is easiest to read; • the background colour of the questionnaire form should not be too strong or bright. Light pastel colours are usually suitable. Background colour should contrast enough with the text to allow for ease of reading; • avoid colour combinations such as red and blue or green and orange. Black on pale yellow is a good combination, but it is difficult to get the right yellow. Testing
Questionnaires
Conclusion It is important to test each of these aspects of questionnaire
design with a group of respondents before finalising the questionnaire. If necessary, the form can then be modified and retested until respondents can complete it accurately and quickly with a minimum of errors. Which of the following occurs when respondents tend to answer questions with a certain slant?
What is recall period in survey?Survey designers must determine the length of the recall period and face a trade-off as increasing the recall period provides more information, but increases the likelihood of recall error. A statistical framework is used to explore this trade-off.
How do you know if a survey is biased?A survey question is biased if it is phrased or formatted in a way that skews people towards a certain answer. Survey question bias also occurs if your questions are hard to understand, making it difficult for customers to answer honestly.
Which term refers to a question that can be answered by gathering data from respondents?A questionnaire is a list of questions or items used to gather data from respondents about their attitudes, experiences, or opinions. Questionnaires can be used to collect quantitative and/or qualitative information.
|