ArticleDesign Effects in the Transition to Web-Based Surveys
Introduction
In the early 1970s telephone technology provided a new and innovative medium to conduct surveys. The telephone quickly became a major survey mode because it was relatively inexpensive and had nearly 90% household coverage rates. In addition, the development of random-digit-dialing procedures allowed efficient random sampling of households, and research indicated that telephone survey results were quite similar to those obtained via face-to-face interviews.1 As telephone surveying advanced, however, methodologists quickly realized that how this new technology was used to conduct surveys was limited by concerns about survey error. Many new techniques were developed to deal with the limitations of the telephone mode (e.g., shortening response scales and converting fully labeled scales to polar point–labeled scales) and a vast body of research addressing the strengths and limitations of telephone surveying has been developed.
Surveyors are now in the beginning stages of another major change in survey methods as cultural changes in how the telephone is viewed and used (e.g., reliance on answering machines) have led to reductions in the advantages that previously drew them to the telephone (i.e., coverage and cooperation rates), and technological innovation has produced a new mode of data collection, the web survey.2 The Internet provides significant opportunities for innovation, making the web survey an incredibly exciting data collection tool. However, as surveyors incorporate web surveys into their major data collection repertoire, it is useful to keep in mind the lessons of the past. Use of the various innovative techniques possible in web surveys should always be mitigated by concern with minimizing survey error, which is now conceptualized as sampling (i.e., smaller sample sizes yield less-precise estimates), coverage (i.e., error resulting from part of the population not having a known, non-zero chance of being sampled), nonresponse (i.e., error resulting from differences between sampled individuals who completed the survey and those who did not), and measurement error (errors resulting from poor question or questionnaire design).2, 3
Each of these sources of error is influenced by the choice of survey mode and by how the survey is designed within a mode; however, the challenge for web surveyors in general and eHealth methodologists in particular is that the body of research addressing how survey error is affected in web surveys is in its infancy. As a result, there is little information about the effects of using techniques developed for use in other modes or of using new web innovations on survey error in web surveys. The research that has been done, however, is instructive. This paper focuses primarily on one of the four sources of survey error—measurement error—and provides an overview of how choosing the web mode affects the asking and answering of questions. In doing so, special attention is paid to the transition from telephone surveys to web surveys, as this transition poses significant challenges because of fundamental differences between these two modes and because of the increasing use of mixed-mode survey designs.
Section snippets
Factors External to Web Survey Design That Affect Measurement
A number of cultural or social challenges affect how respondents interact with web surveys. Many people are wary of the Internet because they feel that they have little control over it. As more people learn about Internet fraud such as “phishing” (i.e., an identity theft scam involving the impersonation of a legitimate business/organization to get one’s personal information), the spread of computer viruses, and the collection of personal information through “spyware,” they become increasingly
What Is Known About Respondent Processing and Response Behavior?
Schwarz4 argued that survey respondents conduct themselves in survey situations as if they are involved in a conversation, with the survey itself representing the researcher’s contribution to the conversation. Respondents abide by universally observed rules of communication that dictate, for example, that one should be understandable, clear, concise, honest, and not repetitive,5 and they expect the researcher to do the same. In applying conversational rules to the survey process, respondents
Formal Feature #1: Question Format
Tradition is a major driving force behind most survey construction. Within a single mode, it is logical to use the question formats that work best, and over time specific question formats and survey modes become coupled and tradition is born. The web is no exception. Despite its youth, web survey traditions have begun to develop as researchers adjust to the absence of an interviewer who could interpret, prompt, cajole, or otherwise motivate respondents to provide complete and accurate answers.
Formal Feature #2: Visual Elements in Web Surveys
Even when the conventional practice of varying question format across telephone and web modes is avoided, other differences that are more difficult to resolve may arise because these survey modes rely on fundamentally different types of communication. In telephone surveys, respondents engage in a conversation, not too unlike those they have every day. The essential form of communication here is aural. The words that respondents hear are their major source of information about what is being
From Single- to Multiple-Column Scalar Presentations
Oftentimes web surveyors take advantage of the horizontal orientation of screens by displaying response options in columns rather than in a single vertical display. Using data from a 2001 paper survey experiment containing the same question and scales presented in Figure 2, Christian and Dillman12 found that respondents were more likely to choose “good” and less likely to choose “very good” when the options appeared in multiple columns. In a web survey, Christian15 found the same significant
Unintentional Effects of Grouping
In web surveys researchers can use visual features to assist respondents; however, the use of such features can cause unintended effects. Smyth et al.16 reported on a series of experiments designed to address the effects of using spacing and headings to subgroup response options in a single-answer question (Figure 3), a practice used in an ongoing national survey. Using data from three web surveys, they reported that subgrouping increased the number of options respondents chose and the
Encouraging Specific Response Formats
Since web survey respondents lack interviewers to convert their answers into acceptable formats, web survey designers often build error messages into their questionnaires to get respondents to correct their mistakes. Such error messages, however, increase respondent frustration and survey termination, so it is necessary to use visual elements to help respondents answer correctly the first time and avoid error warnings. But, as demonstrated above, visual features can have undesirable effects. It
Conclusion
Just as telephone survey practices were constrained and shaped by concern with survey error, the use of the abundant potential for innovation that the Internet provides should also be mitigated by a concern with the four main sources of survey error: sampling, coverage, nonresponse, and measurement. While minimizing total survey error (i.e., the aggregation of all four types of error) is the ultimate goal, the present conversion from a predominantly aural survey mode, the telephone, to a
References (25)
- et al.
Surveys by telephone: a national comparison with personal interviews
(1979) Mail and Internet surveys: the tailored design method
(2000)Research on survey data quality
Public Opin Q
(1987)Cognition and communication: judgmental biases, research methods, and the logic of conversation
(1996)Logic and conversation
- et al.
Formal features of rating scales and the interpretation of question meaning
Int J Public Opin Res
(1998) - et al.
Internet data collection
(2004) - et al.
Comparing check-all and forced-choice question formats in web surveys
Public Opin Q
(2006) - et al.
Does “yes or no” on the telephone mean the same as “check-all-that-apply” on the web?
(2006) Data quality in mail, telephone, and face to face surveys
(1992)
To mix or not to mix: data collection modes in surveys
J Off Stat
The influence of graphical and symbolic language manipulations on responses to self-administered questions
Public Opin Q
Cited by (114)
Variables affecting pharmacy students' pursuit and attainment of postgraduate residency and fellowship positions
2022, Currents in Pharmacy Teaching and LearningCitation Excerpt :The advantages of this approach were reduced survey burden and increased likelihood of survey completion. We utilized a modified tailored design method for web-based surveys to maximize response rate across four contacts.11 This method is a gold standard when conducting survey research and has been used successfully in multiple survey-based studies.
Expatriate family adjustment: How organisational support on international assignments matters
2022, Journal of International ManagementSurvey of practice preferences in deep brain stimulation surgery in the United States
2022, Interdisciplinary Neurosurgery: Advanced Techniques and Case ManagementMilitary Response to Medical Crises - Consensus Recommendations for Military-Civilian Transitions of Care
2023, Disaster Medicine and Public Health Preparedness