Limits and Considerations in Control Groups

Control groups have become an almost required element in trademark surveys. Survey methodology, however, derives from the field of sociology and political science where there was no such concept of 'control' groups. The studies were designed to be descriptive of a phenomenon. As such, the surveys contained no 'controls,' but could still none the less offer useful information. The political polling of today is such an example. Surveys used for intellectual property purposes have been heavily influenced by the field of experimental psychology, from which the concept of the 'control' emanates. The typical understanding of how a control group operates is that one group or cell receives a test stimulus and the other the control stimulus. The control stimulus is similar to the test but absent the alleged infringing aspects. The difference between their results reflects the 'net' confusion level, test minus 'noise.' This definition, however, fails to tell the full story that control groups may not always be needed, and, when needed, may not solve certain aspects of noise or biases merely by their presence.

For a control group to operate effectively, it is best to specify the exact hypothesis that one is positing is causing 'noise' and to design the study to control for that factor explicitly. For example, in a 'squirt study,' where an array of products is shown sequentially, a hypothesis may be that any products shown in such a sequential array would cause a certain level of measured confusion. If this is one's only reason for employing a control, then it would be clear that an 'Eveready study' (which does not present a sequential array) would not require a control. The point is simply that counsel must understand in advance what the hypothesis or hypotheses are for requiring a control group in order to understand whether the control is needed and valid.

The process of evaluating hypotheses is not something specific to the field of marketing research. It is a logical assessment of comparing the following: what is being tested and what type of finding is sought to be relied on from the test. Comparing these two will cause an astute logician to determine whether the test adequately protects the results from having resulted from other potential causes (ie, what is often termed 'noise').

There are, however, some deeper, more subtle but no less important aspects to control groups that are often not discussed in IP law reviews. Below are just two examples out of many. (Greater detail can be found
in reading Cook and Campbell's
1979 book 'Quasi-Experimentation,' Design & Analysis Issues for Field Setting, typical required reading in social science PhD programs.)

A Control for Complex Stimuli May Deduct Not Only 'Noise'

It's often stated that a control stimulus (that stimulus presented to the control group) should be as close to the test stimulus (that stimulus presented to the test group), but absent the alleged infringing element(s). This way, one can presumably parse out the amount of confusion, for example, emanating from the test, while accounting for the percentage of such confusion that would have been measured even without the alleged infringing elements. However for unique stimuli or complex stimuli, such a simple approach can often amount to a futile effort to find the 'closest' control when in fact doing so does not measure noise any more but instead actually is measuring confusion that should be 'counted' and not deducted from the test group. Political polling provides a good example for with people somehow we are attuned to see uniqueness that we often neglect to see for brands. For example, if one is interested in percentages of likely voters for Hillary Clinton in a particular presidential primary, one typically presents the stimulus (the name of the candidate) and asks a series of questions. First off, a control is typically not used because there is no reasonable hypothesis that there is 'noise' (i.e., stating yes or no in any particular biasing way). But what if the purpose of the study was to determine if a particular Hillary look-alike with her picture placed on hang-tags on a particular line of clothing (say pants suits) is causing respondents to be confused (as to source, affiliation or endorsement)? Assume that testing the stimulus with an Eveready approach is the method chosen by the researcher. Show respondents the clothing/hang tag and ask some questions about it. Is there a reasonably hypothesis that something other than the look-alike might be causing confusion to Hillary Clinton? Well yes there is in fact because the 'look' of Hillary Clinton is complex as is the look of any person. Thus, a host of factors might be in part causing the results such as: gender, height, width, height/width ratio, hairstyle, smile, and so on. Deciding in this case to employ a control that would otherwise be the same but absent the alleged infringing elements is almost folly because doing so requires a keen assessment of the factors that make up the 'look' and identity of Hillary Clinton. One can look for or mock up another person on a hang-tag to act as a control stimulus, same gender, perhaps similar height/width, and so on. The conceptually flawed part is the 'and so on.' At what point can one determine that one is removing extraneous factors causing confusion as opposed to Hillary-specific factors. The complexity of the stimulus – and this occurs for trade dress as well not just people – may make a control stimulus likely to contain aspects that should not be 'deducted' as noise.

A Control Often Does Not Remove Noise Attributable to 'Interaction Effects'

The use of a control cell is not a panacea that can 'repair' or 'neutralize' all problems or flaws (such as the use of leading questions or the presence of order bias) that may be present in the questions used in the test cell. In all surveys, it is important to distinguish between 'main effects,' which result from the respondents' exposure to the stimulus that is the subject of the survey, and 'interaction effects,' which result from the interaction between the interviewer and the respondent, including from the questions themselves. Control cells are useful in attempting to measure main effects and to identify causes extraneous to the survey process that could distort the results (such as guessing, intentional wrong answers, preconceived notions of the respondents and the like), which need to be accounted for in analyzing the test cell results. However, control cells cannot address interaction effects caused by flaws in the questions asked and that are introduced by biased questions that could differentially impact the stimuli and observations as between the test cell and the control cell. Such differential impact results in different levels of bias between the test and control cells. For example, if there are multiple questions asked in each cell, the questions following that very first question may be affected by the answers to the prior questions. Thus, an interaction effect may form between cells after such a first question. If a sequence of questions (source, affiliation, authorization) is asked, then the responses to the latter questions might be differentially impacted based on the responses to the source question. If this were to occur, it means that there is an effect not being captured by the 'noise' measure in the control cell. In other words, the noise measure may have taken into account less than what may be needed to parse out noise.

Any observation (question and answer) that follows the first question can be affected not only by the stimuli being tested, but also by that stimulus interacting with question 1 itself. Similarly, any observation (question and answer) that follows the second question can be affected not only by the stimuli being tested but by even more complex interactions (stimulus interacting with question 1 itself (a two-way interaction), and the stimulus interacting with question 1 and 2 together (a three-way interaction).

Conclusion

It is therefore incumbent on the researcher to attempt to eliminate such effects by not having an earlier observation (question and answer) include elements that are biasing and therefore affect a later observation (question and answer). It is also useful for attorneys to ask a priori not 'what should be used as a control' (the typical question), but 'what are the reasonable causes of noise' and 'for what effect should we be attempting to control.' These latter questions will cause a better sense of how to design and then defend a choice of control and questionnaire design.

Dr. Alex Simonson, Ph.D, is president of Simonson Associates, Inc., in Englewood Cliffs, NJ, and has been a survey researcher for more than 15 years. He may be contacted at [email protected], or visit www.simonsonassociates.com.

Control groups have become an almost required element in trademark surveys. Survey methodology, however, derives from the field of sociology and political science where there was no such concept of 'control' groups. The studies were designed to be descriptive of a phenomenon. As such, the surveys contained no 'controls,' but could still none the less offer useful information. The political polling of today is such an example. Surveys used for intellectual property purposes have been heavily influenced by the field of experimental psychology, from which the concept of the 'control' emanates. The typical understanding of how a control group operates is that one group or cell receives a test stimulus and the other the control stimulus. The control stimulus is similar to the test but absent the alleged infringing aspects. The difference between their results reflects the 'net' confusion level, test minus 'noise.' This definition, however, fails to tell the full story that control groups may not always be needed, and, when needed, may not solve certain aspects of noise or biases merely by their presence.

For a control group to operate effectively, it is best to specify the exact hypothesis that one is positing is causing 'noise' and to design the study to control for that factor explicitly. For example, in a 'squirt study,' where an array of products is shown sequentially, a hypothesis may be that any products shown in such a sequential array would cause a certain level of measured confusion. If this is one's only reason for employing a control, then it would be clear that an 'Eveready study' (which does not present a sequential array) would not require a control. The point is simply that counsel must understand in advance what the hypothesis or hypotheses are for requiring a control group in order to understand whether the control is needed and valid.

The process of evaluating hypotheses is not something specific to the field of marketing research. It is a logical assessment of comparing the following: what is being tested and what type of finding is sought to be relied on from the test. Comparing these two will cause an astute logician to determine whether the test adequately protects the results from having resulted from other potential causes (ie, what is often termed 'noise').

There are, however, some deeper, more subtle but no less important aspects to control groups that are often not discussed in IP law reviews. Below are just two examples out of many. (Greater detail can be found
in reading Cook and Campbell's
1979 book 'Quasi-Experimentation,' Design & Analysis Issues for Field Setting, typical required reading in social science PhD programs.)

A Control for Complex Stimuli May Deduct Not Only 'Noise'

It's often stated that a control stimulus (that stimulus presented to the control group) should be as close to the test stimulus (that stimulus presented to the test group), but absent the alleged infringing element(s). This way, one can presumably parse out the amount of confusion, for example, emanating from the test, while accounting for the percentage of such confusion that would have been measured even without the alleged infringing elements. However for unique stimuli or complex stimuli, such a simple approach can often amount to a futile effort to find the 'closest' control when in fact doing so does not measure noise any more but instead actually is measuring confusion that should be 'counted' and not deducted from the test group. Political polling provides a good example for with people somehow we are attuned to see uniqueness that we often neglect to see for brands. For example, if one is interested in percentages of likely voters for Hillary Clinton in a particular presidential primary, one typically presents the stimulus (the name of the candidate) and asks a series of questions. First off, a control is typically not used because there is no reasonable hypothesis that there is 'noise' (i.e., stating yes or no in any particular biasing way). But what if the purpose of the study was to determine if a particular Hillary look-alike with her picture placed on hang-tags on a particular line of clothing (say pants suits) is causing respondents to be confused (as to source, affiliation or endorsement)? Assume that testing the stimulus with an Eveready approach is the method chosen by the researcher. Show respondents the clothing/hang tag and ask some questions about it. Is there a reasonably hypothesis that something other than the look-alike might be causing confusion to Hillary Clinton? Well yes there is in fact because the 'look' of Hillary Clinton is complex as is the look of any person. Thus, a host of factors might be in part causing the results such as: gender, height, width, height/width ratio, hairstyle, smile, and so on. Deciding in this case to employ a control that would otherwise be the same but absent the alleged infringing elements is almost folly because doing so requires a keen assessment of the factors that make up the 'look' and identity of Hillary Clinton. One can look for or mock up another person on a hang-tag to act as a control stimulus, same gender, perhaps similar height/width, and so on. The conceptually flawed part is the 'and so on.' At what point can one determine that one is removing extraneous factors causing confusion as opposed to Hillary-specific factors. The complexity of the stimulus – and this occurs for trade dress as well not just people – may make a control stimulus likely to contain aspects that should not be 'deducted' as noise.

A Control Often Does Not Remove Noise Attributable to 'Interaction Effects'

The use of a control cell is not a panacea that can 'repair' or 'neutralize' all problems or flaws (such as the use of leading questions or the presence of order bias) that may be present in the questions used in the test cell. In all surveys, it is important to distinguish between 'main effects,' which result from the respondents' exposure to the stimulus that is the subject of the survey, and 'interaction effects,' which result from the interaction between the interviewer and the respondent, including from the questions themselves. Control cells are useful in attempting to measure main effects and to identify causes extraneous to the survey process that could distort the results (such as guessing, intentional wrong answers, preconceived notions of the respondents and the like), which need to be accounted for in analyzing the test cell results. However, control cells cannot address interaction effects caused by flaws in the questions asked and that are introduced by biased questions that could differentially impact the stimuli and observations as between the test cell and the control cell. Such differential impact results in different levels of bias between the test and control cells. For example, if there are multiple questions asked in each cell, the questions following that very first question may be affected by the answers to the prior questions. Thus, an interaction effect may form between cells after such a first question. If a sequence of questions (source, affiliation, authorization) is asked, then the responses to the latter questions might be differentially impacted based on the responses to the source question. If this were to occur, it means that there is an effect not being captured by the 'noise' measure in the control cell. In other words, the noise measure may have taken into account less than what may be needed to parse out noise.

Any observation (question and answer) that follows the first question can be affected not only by the stimuli being tested, but also by that stimulus interacting with question 1 itself. Similarly, any observation (question and answer) that follows the second question can be affected not only by the stimuli being tested but by even more complex interactions (stimulus interacting with question 1 itself (a two-way interaction), and the stimulus interacting with question 1 and 2 together (a three-way interaction).

Conclusion

It is therefore incumbent on the researcher to attempt to eliminate such effects by not having an earlier observation (question and answer) include elements that are biasing and therefore affect a later observation (question and answer). It is also useful for attorneys to ask a priori not 'what should be used as a control' (the typical question), but 'what are the reasonable causes of noise' and 'for what effect should we be attempting to control.' These latter questions will cause a better sense of how to design and then defend a choice of control and questionnaire design.

Dr. Alex Simonson, Ph.D, is president of Simonson Associates, Inc., in Englewood Cliffs, NJ, and has been a survey researcher for more than 15 years. He may be contacted at [email protected], or visit www.simonsonassociates.com.