Survey design is the process of preparing a complete plan of operations to be followed in conducting a survey and disseminating its intended results.
Specifically, it includes, among others, decisions on such factors as variables to be included in the survey (called survey variables), the method of data collection (whether by direct interview, telephone interview, or self-administered questionnaire), construction of questionnaire, organizing fieldwork, management of non-sampling error, data processing, and data analysis.
It seems obvious that the survey objectives covered under the survey design determine the sample design. In practice, the sample design must be developed as an integral part of the overall survey design.
Survey design and sample design are thus the two interrelated concepts, and one is complementary to the other. There are several decisions to be made in obtaining a sample from a population. Each requires unique information.
Here we raise a few questions that must be addressed in the process of obtaining a sample.
For sampling to be representative, the population from which the sample has to be drawn is to be defined unambiguously.
No confusion should arise about whether the population consists of individuals, households, or families, or a combination of these.
The objectives of the survey need to be clearly stated. A clear statement helps to select appropriate study design, to decide on the size of the sample, and to prepare suitable survey instruments.
Parameters of Interest
Population parameters are summary descriptors (e.g., mean, variance, proportion, prevalence, etc.) of variables of interest in the population.
Statistics are descriptors of relevant variables computed from the sample data.
They are estimators of population parameters. Sample statistics serve as the basis of our inference about the population because they are the best estimates of the population parameters.
The sampling frame is the list of population units from which the sample is drawn.
The theory of probability sampling is heavily dependent on the existence of a good sampling frame. Ideally, a sampling frame is a list of units or groups of units of the population to be sampled.
To be useful; a sampling frame should be organized and arranged in such a manner that every unit occurs once and only once in the list, and no unit is excluded from the list.
Type of Sample
Whether a probability sample or a non-probability sample is to be adopted, must be decided in advance. If a sampling frame is available, a probability sample is feasible, if the resource permits.
Size of the sample
One of the most important problems in planning a sample survey is that of determining how large a sample is needed for the estimates to be reliable enough. The decision is important for several reasons.
Too large a sample involves huge cost, additional manpower, more materials, and prolonged time, while too small a sample invalidates the results. Then the question is: what is the optimum size of the sample?
The absolute size of a sample is much more important than its relative size. The eventual sample size is usually a compromise between what is ‘desirable’ and what is ‘feasible.’
Cost considerations influence decisions about the size and type of sample and also the data collection methods. Almost all studies have some budgetary constraints, and this can lead to the use of a non-probability sample.
The probability sample incurs call-back costs, listing costs, and a variety of other costs that are not necessary when a non-probability sampling scheme is adopted.
The following document is intended as a brief guide to facilitate survey design and development systematically. The document consists of four sections that cover the main stages of survey development and administration.
Each section has a series of questions intended as a guide to facilitate addressing main issues. (Adapted from David Hall, Ph.D. for ACHRN)
Section I: Survey Groundwork
- Do you have an external advisory group to assist you in the whole survey work? Non-experts benefit from expert advice and experts benefit from peer review.
- Do you need ethical approval? (if in doubt-ask if you want to publish the results in peer-reviewed journals, they may require ethical approval). What arrangements will be made to get ethical approval?
- What are the questions you wish to answer? What are the specific objectives of the survey to address these questions? Consider:
- What new information will this survey provide?
- What policy, program, funding, or other kinds of decisions will be made based on the survey findings?
- What data are relevant to the research questions, hypotheses, and decisions? What are the possible sources of these data in addition to the survey designed for the study in question?
- What are the geographical/political/demographic domains for the investigation? What might be the implications of each of these for the interpretation of the results and the application of the findings?
- It is very seldom true that a questionnaire is the only means available to gather data. What are other possible means of gathering relevant data to address your questions? What are the pros and cons of conducting this survey compared to other possible means of data collection?
- What type of design will be used for the project? Do you want descriptive information, or do you want to compare groups? Have you arranged for help with the design and planning phase?
- What type of survey will be conducted (e.g., telephone survey, self-administered, or face to face interview)? Consider the advantages and disadvantages of each survey method. Why will this type of survey be chosen over another type?
- Has there been a thorough search and review of other relevant studies? Have you looked for existing tools with known reliability and validity?
- Have you defined the target population for your survey? How will this target population be sampled (e.g., simple, stratified, systematic, focus group, convenience, or snowball)?
- How many responses are needed? What steps will be taken to increase the response rate? How will a low response rate be handled? What kind of follow-up contacts might be taken to increase returns? How does your follow-up plan fit with your promise of confidentiality and anonymity?]
- Have you developed a timeline with deadlines and contingencies for piloting, revisions, and possible delays? Have you allowed time for administrative review, ethics approval (if needed), and inevitable delays? Have you shared this timeline with stakeholders?
- What staffing will be needed to collect the responses, enter and analyze the data, and write the report? Who will these people be (staff, contract positions)?
- What potential criticisms might be made of the survey and its administration? For example, people sometimes argue: “The sample was not representative, because it didn’t include any of us.” Which criticisms might be plausible? How will you handle each of these possible criticisms?
Other points pertinent to the groundwork, include, among others, relevance, feasibility, applicability, and urgency of the data needed.
Section II: Instrument Development
Before beginning the development of the questionnaire/interview schedules, the objective of the survey must be very clearly articulated. Each question on the survey must relate to the objectives.
The analysis for each question must be considered, as each question is designed.
- How does each question meet its objective? Does each provide information that is not otherwise available?
- Can you use questions developed and tested in another project? If you intend to compare responses from two sources, are the questions the same?
- Which question format are you going to use to collect data? Do you want numeric information, text information, or both?
- For numeric questions, what type of data will be most useful? Do you need nominal, ordinal, interval, or ratio data for your planned analysis and tables? Do you want to leave space for comments?
- Consider the possible responses to each question. Are response categories clear and mutually exclusive?
- For each question, what might a non-response mean? How will non-responses be handled?
- Are any questions going to be open-ended? What strategy will you use to analyze open-ended data?
- At each draft, consider the questionnaire as a whole. Are the questions categorized in sections or sub-groups (e.g., demographic information, preferences, etc.)? Can the survey questions be organized in a better way to improve flow and maximize responses? For example, it is usually best to place sensitive questions at the end of a section rather than at the beginning of a survey.
- Have the questions been verified by a group of content experts for clarity, intent, and appropriateness?
- Have all important concepts and terms been given good working definitions that are clear and easy to understand? Has a naive layperson read and understood them?
- Is the reading level appropriate for the group of content experts for clarity, intent, and appropriateness?
- How long is your questionnaire? Does it look like it will take a long time or be difficult to do? Can any questions be removed? Are there lots of blank space, particularly on the first page? Or the font large enough for the participants to read? Are all the pages numbered?
- Have the questions been carefully sequenced? Ordered? Have the skipping patterns been checked for consistency?
- Did you make sure that there aren’t any spelling errors or grammatical errors in the questionnaires?
- How will the questionnaire/interview schedule be piloted? How many people need to be in the pilot?
Section III: Conducting the Survey
- Does the cover page convey the rationale and objectives of the survey clearly and succinctly so that the target group can easily understand them? Does it make them want to complete the survey?
- Does the cover page indicate who is responsible for the survey, why and how the information will be used, and who to contact for more information?
- Does the cover page clearly state how issues associate with confidentiality and anonymity are handled? Does it indicate what will be done with the results?
- How are the participants’ informed consent obtained?
- For questionnaires, are there clear instructions? Do the instructions indicate how long it takes to complete? How to complete the questions, and what to do with the completed forms?
- If you are using interviews, are interviewers well trained to ensure that the data are reliable and valid?
- What will be done to help people with language barriers or disabilities or any other shortcomings?
- How will participants be able to access the results of the survey?
Section IV: Dealing with Data
Before you start:
- Consider how the data are to be reported. Prepare sample tables and graphs for numeric data.
- What information will be available? What information will not be available?
- Consider possible response patterns. What decisions would be made based on each possible pattern?
- What are additional data available to supplement your data? How much these data support, confirm, or contradict your findings?
- What possible weaknesses might there be in the data that could compromise strong conclusions?
- What might additional information be needed to address these weaknesses?
Once you have the data:
- How will the response data be entered? For interviews, will you collect tape recordings or interviewers’ notes? How will you deal with tape recordings? (Transcriptions? Time required? Cost?)
- Do you have individuals skilled in using the computer analysis packages needed?
- How will the data be analyzed? Do you have access to others to help with the analyses? How long will it take to do the analyses?
- For numeric data, how will missing cases be handled? (For example, will the record be deleted? Will the entry be coded as ‘O’?
- How will the data be handled once the survey and final reports are completed? What are the arrangements for archiving questionnaires?
Section V: Disseminating the Results
- Who will receive the results of the survey? What is the most appropriate method to present the results? Written reports, personal presentation, a telephone conference?
- Will different forms of the presentation be prepared for the stakeholders and the public?
- If there is a final report, does it include all the elements of the survey, including a copy of the survey instruments?
- Is the report easily readable? Is it written and at the language level of the reader?
- Is the survey going to be published in newsletters, annual reports, and other publications? (If the survey is going to be published in a peer-reviewed journal, ethical approval may be one condition for publication).