Online Interviewing for Use in Lanham Act Litigation

Internet or online interviewing is a method that can provide reliable and valid survey data. Often one sees or hears of Internet surveys where respondents are recruited haphazardly online (at news sites for example), and such methods often cast long shadows as to the scientific nature and validity of Internet surveys for purposes of litigation. The method of Internet interviewing, however, is not inherently flawed or somehow tainted by virtue of some surveys that have used improper or unreliable methods.

Evolution of the Internet

The Internet has evolved over the past five years in significant ways, causing interviewing methods, if conducted well, to be controlled and reliable. First, the panels from which respondents are typically recruited have evolved into being representative of the online population and fairly representative of the U.S. population as a whole, much more so than the typical mall-intercept interview. This is due to numerous factors, among them the method of recruitment via Internet panels that allows us also to weight e-mail invitations by population proportions or other demographic characteristics. We can approach a representative sample of a particular defined universe.

Moreover, the Internet has penetrated a significant proportion of the U.S. population (over 70%) (Pew Internet & American Life Project Surveys, March 2000-April 2006; and Nielsen Net Ratings, May 2007). Five years ago, Internet penetration was about 30%, a sharp contrast. Compare this with mall-intercepts, where, of all regional malls in the United States, those that we have access to (on-site facilities for allowing interviewing) account for significantly less than 70% of the population. (This is derived via calculations I made using information on the total number of enclosed (regional) shopping malls in the United States (National Research Bureau's 2000 CD-ROM edition) compared with those listed in the American Marketing Association Greenbook, 2000-2007 as containing interviewing facilities and the U.S. Bureau of the Census (Census 2000).) Contrast this also with the typical mall-intercept, where, at most, typically we interview respondents in about 12 markets for one particular survey. Mall diversity among eight or 12 locations cannot match that of choosing respondents from around the United States among Internet panels. This is because Internet methodology is significantly geographically diverse (mirroring telephone interviewing) in finding people for a survey from literally across the entire United States. As mobility and Internet penetration rises, our ability to find and stop people in malls may not be as optimal as finding people wherever they happen to be, so long as they're able to have access to the Internet, whether at home, work, or a coffee shop. More importantly, perhaps, central location interviewing, like mall intercepts, is not statistically projectable, is deemed 'convenience' samples, and thus does not possess any more reliability. Its strength lies mainly in having been the only viable alternative to find people and show them stimuli when door-to-door became unfeasible.

Advantages of Computer-Based Interviewing

The advantages of computer-based interviewing (whether on the Internet or via CAPI (Computer-Assisted Personal Interviewing) over malls) are significant. These include: a) eliminating interviewer errors such as missing answers, failed skip patterns (asking the wrong questions, not asking the right questions), erasures, not properly rotating questions or stimuli, filling information in later if missed, failing to show stimuli in the manner indicated in the instructions, showing test instead of control or vice-versa; b) eliminating transcription from paper to computer and thereby eliminating errors that can creep in through the process; and c) making the coding process more consistent via having all responses in sight in one file as one codes, thereby reducing the inconsistency in coding that can occur when coding from multiple physical questionnaires. The time in the field is reduced, and the costs are often lower. One can monitor the progress readily and routinely as opposed to having to mail or express questionnaires back from the field to review data on an ongoing basis.

Other advantages are also ironically in the part most observers find uncontrolled, in the self-administration process. For example, sometimes researchers ask a question 'does or doesn't ' ' as opposed to 'does ' ' with the assumption that perhaps asking it the former way produces an evenhanded approach to reduce demand biases. An alternative is to say 'does ' ' and follow the question with stating explicitly to the respondent as part of the question, 'yes, no, or are you not sure' ' this supplies the respondent with the balance. Though nobody I know of has produced evidence of such biases when using the phrase 'does ',' one can bypass the argument with the self-administration process because the answer choices (yes, no, and don't know or not sure) present themselves clearly below the question. There is an immediate visual cue to the respondent that there are balanced responses possible for the respondent to choose among. This applies equally to the phrase sometimes added awkwardly in mall-intercepts, 'if you don't know, please just tell me so.' On the Internet, the 'don't know' choice is as prominent as is the 'yes' or 'no' choices, eliminating the need for awkward and sometimes even confusing cues.

Significant strides have been made in stimuli presentation via computer that under any circumstances simply cannot be reproduced in standard pen-paper mall-intercepts. Visual simulations of entire shelf arrays, simulating approaching the stimuli from afar moving in to a closer field (often useful to test for initial interest confusion), full rotations of stimuli (left to right, top to bottom, size rotations, SKU rotations), timing respondents' answers (useful in dilution studies), and other visual and aural cues are robust in Internet interviewing.

Control Processes

That said, significant control pro- cesses are required when implementing Internet studies. Without these, Internet studies can be seriously flawed and unreliable. These controls were often not put into place five years ago and are not still by many researchers who do not engage in ongoing Internet research. Among these controls are the following: eliminating the 'previous' button in typical Internet studies to avoid respondents' changing answers after seeing something else in the survey; stopping the interview if someone presses the Web browser's 'back' button; activating an e-mail link to be used once and only once and deactivating it once the survey has been opened even at its initial page; conducting validation via telephone to confirm that the survey took place with the person who represented to us that he/she took the survey and that he/she qualified for the survey ' this particularly has been very difficult administratively to convince the panel owners to allow to ensure evidentiary reliability of the data; covering stimuli until fully loaded, loading times varying based on computer connection speeds; setting of stimuli to be visually similar on various screen sizes; preservation of the data file from the hosting facility to ensure no modifications to the data have occurred, among others.

The Internet is still limited by the fact that we are not there to observe the process. (This will likely change over the next decade as more OEM computers will be fitted with cameras and we will be able to watch the survey over Web-based cameras to ensure it is being taken properly.) However, as of now, the reality that we don't 'see' the respondent during administration lends itself to speculation that perhaps the respondent talked to others during the process; perhaps he/she took a long break and returned only after learning something about the questions at issue; perhaps the respondent didn't fully pay attention during the process. These are questions that are not unique to Internet interviewing, but derive from the process of self-administration at a remote location. There are ways to mitigate these concerns, like calling (random digit dial telephone interviewing) and having the interviewer merely be a person on the line while the respondent takes the interview. There are other ways as well, such as timing out a respondent after a set time that seems unusually long, like a minute. Also, looking at completion times (recorded in the log) can be used to determine if there are outliers in terms of taking the survey, perhaps indicating having left the scene to do something else in the process.

Consider, though, the tried and true method of the telephone-mail-telephone interview. Here, we recruit via telephone; we mail the respondent a stimulus or stimuli, and then we conduct the interview on the phone. This very common method (both in strategic marketing research and for litigation) suffers from the limitation more severely, because when we ask the respondent to look at a stimulus and then return it to the envelope prior to questioning, we don't know if he/she has actually done so. In the case of the Internet, we can know whether respondents have proceeded by having something on the next page that covers the stimulus and exhibits a code for the respondent to read off or type in. But the telephone-mail-telephone method is not excluded as being unreliable nor should it be. The argument of rampant respondent failures to follow simple instructions is unfounded and speculative.

Conclusion

No method is perfect and no method is tried and true until courts and attorneys are willing to maintain an open mind and thereby benefit from new technologies. While mall intercepts took the place of door-to-door interviewing, Internet interviewing will undoubtedly become the norm over the next decade. Being familiar with the ways to enhance its reliability and validity will be necessary to create scientifically valid, controlled, and reliable studies.

Alex Simonson, J.D., Ph.D., is President of Simonson Associates, Inc., in Englewood Cliffs, NJ, and has been a survey researcher for more than 15 years. He may be contacted at [email protected].

Internet or online interviewing is a method that can provide reliable and valid survey data. Often one sees or hears of Internet surveys where respondents are recruited haphazardly online (at news sites for example), and such methods often cast long shadows as to the scientific nature and validity of Internet surveys for purposes of litigation. The method of Internet interviewing, however, is not inherently flawed or somehow tainted by virtue of some surveys that have used improper or unreliable methods.

Evolution of the Internet

The Internet has evolved over the past five years in significant ways, causing interviewing methods, if conducted well, to be controlled and reliable. First, the panels from which respondents are typically recruited have evolved into being representative of the online population and fairly representative of the U.S. population as a whole, much more so than the typical mall-intercept interview. This is due to numerous factors, among them the method of recruitment via Internet panels that allows us also to weight e-mail invitations by population proportions or other demographic characteristics. We can approach a representative sample of a particular defined universe.

Moreover, the Internet has penetrated a significant proportion of the U.S. population (over 70%) (Pew Internet & American Life Project Surveys, March 2000-April 2006; and Nielsen Net Ratings, May 2007). Five years ago, Internet penetration was about 30%, a sharp contrast. Compare this with mall-intercepts, where, of all regional malls in the United States, those that we have access to (on-site facilities for allowing interviewing) account for significantly less than 70% of the population. (This is derived via calculations I made using information on the total number of enclosed (regional) shopping malls in the United States (National Research Bureau's 2000 CD-ROM edition) compared with those listed in the American Marketing Association Greenbook, 2000-2007 as containing interviewing facilities and the U.S. Bureau of the Census (Census 2000).) Contrast this also with the typical mall-intercept, where, at most, typically we interview respondents in about 12 markets for one particular survey. Mall diversity among eight or 12 locations cannot match that of choosing respondents from around the United States among Internet panels. This is because Internet methodology is significantly geographically diverse (mirroring telephone interviewing) in finding people for a survey from literally across the entire United States. As mobility and Internet penetration rises, our ability to find and stop people in malls may not be as optimal as finding people wherever they happen to be, so long as they're able to have access to the Internet, whether at home, work, or a coffee shop. More importantly, perhaps, central location interviewing, like mall intercepts, is not statistically projectable, is deemed 'convenience' samples, and thus does not possess any more reliability. Its strength lies mainly in having been the only viable alternative to find people and show them stimuli when door-to-door became unfeasible.

Advantages of Computer-Based Interviewing

The advantages of computer-based interviewing (whether on the Internet or via CAPI (Computer-Assisted Personal Interviewing) over malls) are significant. These include: a) eliminating interviewer errors such as missing answers, failed skip patterns (asking the wrong questions, not asking the right questions), erasures, not properly rotating questions or stimuli, filling information in later if missed, failing to show stimuli in the manner indicated in the instructions, showing test instead of control or vice-versa; b) eliminating transcription from paper to computer and thereby eliminating errors that can creep in through the process; and c) making the coding process more consistent via having all responses in sight in one file as one codes, thereby reducing the inconsistency in coding that can occur when coding from multiple physical questionnaires. The time in the field is reduced, and the costs are often lower. One can monitor the progress readily and routinely as opposed to having to mail or express questionnaires back from the field to review data on an ongoing basis.

Other advantages are also ironically in the part most observers find uncontrolled, in the self-administration process. For example, sometimes researchers ask a question 'does or doesn't ' ' as opposed to 'does ' ' with the assumption that perhaps asking it the former way produces an evenhanded approach to reduce demand biases. An alternative is to say 'does ' ' and follow the question with stating explicitly to the respondent as part of the question, 'yes, no, or are you not sure' ' this supplies the respondent with the balance. Though nobody I know of has produced evidence of such biases when using the phrase 'does ',' one can bypass the argument with the self-administration process because the answer choices (yes, no, and don't know or not sure) present themselves clearly below the question. There is an immediate visual cue to the respondent that there are balanced responses possible for the respondent to choose among. This applies equally to the phrase sometimes added awkwardly in mall-intercepts, 'if you don't know, please just tell me so.' On the Internet, the 'don't know' choice is as prominent as is the 'yes' or 'no' choices, eliminating the need for awkward and sometimes even confusing cues.

Significant strides have been made in stimuli presentation via computer that under any circumstances simply cannot be reproduced in standard pen-paper mall-intercepts. Visual simulations of entire shelf arrays, simulating approaching the stimuli from afar moving in to a closer field (often useful to test for initial interest confusion), full rotations of stimuli (left to right, top to bottom, size rotations, SKU rotations), timing respondents' answers (useful in dilution studies), and other visual and aural cues are robust in Internet interviewing.

Control Processes

That said, significant control pro- cesses are required when implementing Internet studies. Without these, Internet studies can be seriously flawed and unreliable. These controls were often not put into place five years ago and are not still by many researchers who do not engage in ongoing Internet research. Among these controls are the following: eliminating the 'previous' button in typical Internet studies to avoid respondents' changing answers after seeing something else in the survey; stopping the interview if someone presses the Web browser's 'back' button; activating an e-mail link to be used once and only once and deactivating it once the survey has been opened even at its initial page; conducting validation via telephone to confirm that the survey took place with the person who represented to us that he/she took the survey and that he/she qualified for the survey ' this particularly has been very difficult administratively to convince the panel owners to allow to ensure evidentiary reliability of the data; covering stimuli until fully loaded, loading times varying based on computer connection speeds; setting of stimuli to be visually similar on various screen sizes; preservation of the data file from the hosting facility to ensure no modifications to the data have occurred, among others.

The Internet is still limited by the fact that we are not there to observe the process. (This will likely change over the next decade as more OEM computers will be fitted with cameras and we will be able to watch the survey over Web-based cameras to ensure it is being taken properly.) However, as of now, the reality that we don't 'see' the respondent during administration lends itself to speculation that perhaps the respondent talked to others during the process; perhaps he/she took a long break and returned only after learning something about the questions at issue; perhaps the respondent didn't fully pay attention during the process. These are questions that are not unique to Internet interviewing, but derive from the process of self-administration at a remote location. There are ways to mitigate these concerns, like calling (random digit dial telephone interviewing) and having the interviewer merely be a person on the line while the respondent takes the interview. There are other ways as well, such as timing out a respondent after a set time that seems unusually long, like a minute. Also, looking at completion times (recorded in the log) can be used to determine if there are outliers in terms of taking the survey, perhaps indicating having left the scene to do something else in the process.

Consider, though, the tried and true method of the telephone-mail-telephone interview. Here, we recruit via telephone; we mail the respondent a stimulus or stimuli, and then we conduct the interview on the phone. This very common method (both in strategic marketing research and for litigation) suffers from the limitation more severely, because when we ask the respondent to look at a stimulus and then return it to the envelope prior to questioning, we don't know if he/she has actually done so. In the case of the Internet, we can know whether respondents have proceeded by having something on the next page that covers the stimulus and exhibits a code for the respondent to read off or type in. But the telephone-mail-telephone method is not excluded as being unreliable nor should it be. The argument of rampant respondent failures to follow simple instructions is unfounded and speculative.

Conclusion

No method is perfect and no method is tried and true until courts and attorneys are willing to maintain an open mind and thereby benefit from new technologies. While mall intercepts took the place of door-to-door interviewing, Internet interviewing will undoubtedly become the norm over the next decade. Being familiar with the ways to enhance its reliability and validity will be necessary to create scientifically valid, controlled, and reliable studies.

Alex Simonson, J.D., Ph.D., is President of Simonson Associates, Inc., in Englewood Cliffs, NJ, and has been a survey researcher for more than 15 years. He may be contacted at [email protected].